2024-03-12T03:16:42.874 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-12T03:16:42.874 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883 --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --owner scheduled_teuthology@teuthology --verbose -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/orig.config.yaml 2024-03-12T03:16:42.920 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T03:16:43.002 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591883' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi098.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBORZAmOWZtokeQB+CMilfY5yhRq4Gplc3CKGrMPgUPGx9q7vU9VWplWnjhKre61wRsoBsHOmSZEIJNdERvKSIPU= smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBS8D6UZq9/Mh1qPo4j827Ceqdn5bL1PWvfMLfF1ULEefVUd9uWOx65j2krI3URLO/cZi1Ijd7i7L0lIaNMOx8= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-12T03:16:43.002 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-12T03:16:43.003 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-12T03:16:43.003 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-12T03:16:43.004 INFO:teuthology.task.internal:Checking packages... 2024-03-12T03:16:43.027 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-12T03:16:43.028 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-12T03:16:43.028 INFO:teuthology.packaging:ref: None 2024-03-12T03:16:43.028 INFO:teuthology.packaging:tag: None 2024-03-12T03:16:43.028 INFO:teuthology.packaging:branch: main 2024-03-12T03:16:43.028 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:16:43.028 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-12T03:16:43.262 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-12T03:16:43.264 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-12T03:16:43.271 INFO:teuthology.task.internal:no buildpackages task found 2024-03-12T03:16:43.271 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-12T03:16:43.312 INFO:teuthology.task.internal:Saving configuration 2024-03-12T03:16:43.329 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-12T03:16:43.335 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T03:16:43.362 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:11:37.565672', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBORZAmOWZtokeQB+CMilfY5yhRq4Gplc3CKGrMPgUPGx9q7vU9VWplWnjhKre61wRsoBsHOmSZEIJNdERvKSIPU='} 2024-03-12T03:16:43.386 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:11:37.564412', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBS8D6UZq9/Mh1qPo4j827Ceqdn5bL1PWvfMLfF1ULEefVUd9uWOx65j2krI3URLO/cZi1Ijd7i7L0lIaNMOx8='} 2024-03-12T03:16:43.386 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-12T03:16:43.392 INFO:teuthology.task.internal:roles: ubuntu@smithi098.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-12T03:16:43.393 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-12T03:16:43.393 INFO:teuthology.run_tasks:Running task console_log... 2024-03-12T03:16:43.473 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f40c16400d0>, signals=[15]) 2024-03-12T03:16:43.474 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-12T03:16:43.483 INFO:teuthology.task.internal:Opening connections... 2024-03-12T03:16:43.483 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi098.front.sepia.ceph.com 2024-03-12T03:16:43.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:16:43.588 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-03-12T03:16:43.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:16:43.676 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-12T03:16:43.683 DEBUG:teuthology.orchestra.run.smithi098:> uname -m 2024-03-12T03:16:43.700 INFO:teuthology.orchestra.run.smithi098.stdout:x86_64 2024-03-12T03:16:43.700 DEBUG:teuthology.orchestra.run.smithi098:> cat /etc/os-release 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:NAME="CentOS Stream" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION="9" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:ID="centos" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:VERSION_ID="9" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:ANSI_COLOR="0;31" 2024-03-12T03:16:43.756 INFO:teuthology.orchestra.run.smithi098.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:16:43.757 INFO:teuthology.orchestra.run.smithi098.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:16:43.757 INFO:teuthology.orchestra.run.smithi098.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:16:43.757 INFO:teuthology.orchestra.run.smithi098.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:16:43.757 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:16:43.757 INFO:teuthology.orchestra.run.smithi098.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:16:43.757 INFO:teuthology.lock.ops:Updating smithi098.front.sepia.ceph.com on lock server 2024-03-12T03:16:43.784 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-03-12T03:16:43.801 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-03-12T03:16:43.801 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-03-12T03:16:43.858 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-03-12T03:16:43.858 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-03-12T03:16:43.858 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-12T03:16:43.859 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-12T03:16:43.859 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-03-12T03:16:43.883 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-12T03:16:43.893 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-12T03:16:43.912 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-12T03:16:43.912 DEBUG:teuthology.orchestra.run.smithi098:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:16:43.914 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-03-12T03:16:43.930 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-12T03:16:43.936 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-12T03:16:43.936 DEBUG:teuthology.orchestra.run.smithi098:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:16:43.971 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-03-12T03:16:44.080 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-12T03:16:44.132 INFO:teuthology.run_tasks:Running task kernel... 2024-03-12T03:16:44.147 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-12T03:16:44.147 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-12T03:16:44.147 DEBUG:teuthology.orchestra.run.smithi098:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:16:44.147 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-12T03:16:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:16:44.163 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-12T03:16:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:16:44.164 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-12T03:16:44.220 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:16:44.221 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-361.el9.x86_64 2024-03-12T03:16:44.221 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-03-12T03:16:44.222 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-361.el9.x86_64 2024-03-12T03:16:44.222 INFO:teuthology.task.kernel:Running kernel on smithi098: 5.14.0-361.el9.x86_64 2024-03-12T03:16:44.222 DEBUG:teuthology.orchestra.run.smithi098:> sudo yum install -y kernel 2024-03-12T03:16:46.408 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-12T03:16:46.517 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-12T03:16:47.850 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-12T03:16:47.933 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-12T03:16:48.957 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-12T03:16:54.525 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 4.2 MB/s | 19 MB 00:04 2024-03-12T03:17:06.774 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 712 kB/s | 6.1 MB 00:08 2024-03-12T03:17:46.020 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 409 B/s | 15 kB 00:38 2024-03-12T03:17:47.981 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-12T03:17:52.879 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-12T03:17:53.064 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:01 ago on Tue 12 Mar 2024 03:17:52 AM UTC. 2024-03-12T03:17:55.052 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:17:55.054 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:17:55.133 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:17:55.144 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:17:55.145 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:17:55.145 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:17:55.145 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-03-12T03:17:55.145 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-03-12T03:17:55.145 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-03-12T03:17:55.145 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:17:55.146 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 100 M 2024-03-12T03:17:55.146 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 123 M 2024-03-12T03:17:55.146 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-03-12T03:18:13.403 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 339 kB/s | 6.0 MB 00:18 2024-03-12T03:18:44.449 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 434 kB/s | 21 MB 00:49 2024-03-12T03:19:00.934 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - BaseOS 62 kB/s | 8.0 MB 02:12 2024-03-12T03:19:06.196 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 568 kB/s | 39 MB 01:10 2024-03-12T03:19:22.336 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 496 kB/s | 33 MB 01:08 2024-03-12T03:19:22.336 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:19:22.336 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.1 MB/s | 100 MB 01:27 2024-03-12T03:19:23.215 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-03-12T03:19:23.602 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-03-12T03:19:23.603 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-03-12T03:19:25.266 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-03-12T03:19:25.266 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-03-12T03:19:27.975 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-03-12T03:19:28.618 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:19:28.752 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:19:30.398 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - AppStream 686 kB/s | 19 MB 00:28 2024-03-12T03:19:31.547 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:19:31.721 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:19:36.422 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:19:36.622 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:19:39.782 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - CRB 1.0 MB/s | 6.1 MB 00:05 2024-03-12T03:19:41.308 INFO:teuthology.orchestra.run.smithi098.stdout:CentOS Stream 9 - Extras packages 42 kB/s | 15 kB 00:00 2024-03-12T03:19:41.359 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:19:42.059 INFO:teuthology.orchestra.run.smithi098.stdout:Extra Packages for Enterprise Linux 32 MB/s | 21 MB 00:00 2024-03-12T03:19:47.059 INFO:teuthology.orchestra.run.smithi098.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-12T03:19:49.249 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-12T03:19:49.250 INFO:teuthology.orchestra.run.smithi098.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-12T03:19:49.342 INFO:teuthology.orchestra.run.smithi098.stdout:Dependencies resolved. 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: Package Architecture Version Repository Size 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout:Installing: 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout:Installing dependencies: 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-12T03:19:49.348 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-12T03:19:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:19:49.349 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction Summary 2024-03-12T03:19:49.349 INFO:teuthology.orchestra.run.smithi098.stdout:================================================================================ 2024-03-12T03:19:49.349 INFO:teuthology.orchestra.run.smithi098.stdout:Install 4 Packages 2024-03-12T03:19:49.349 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:19:49.350 INFO:teuthology.orchestra.run.smithi098.stdout:Total download size: 100 M 2024-03-12T03:19:49.350 INFO:teuthology.orchestra.run.smithi098.stdout:Installed size: 123 M 2024-03-12T03:19:49.350 INFO:teuthology.orchestra.run.smithi098.stdout:Downloading Packages: 2024-03-12T03:19:50.336 INFO:teuthology.orchestra.run.smithi098.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 6.5 MB/s | 6.0 MB 00:00 2024-03-12T03:19:52.411 INFO:teuthology.orchestra.run.smithi098.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 7.0 MB/s | 21 MB 00:02 2024-03-12T03:19:54.613 INFO:teuthology.orchestra.run.smithi098.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 7.6 MB/s | 39 MB 00:05 2024-03-12T03:19:58.130 INFO:teuthology.orchestra.run.smithi098.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 4.3 MB/s | 33 MB 00:07 2024-03-12T03:19:58.130 INFO:teuthology.orchestra.run.smithi098.stdout:-------------------------------------------------------------------------------- 2024-03-12T03:19:58.130 INFO:teuthology.orchestra.run.smithi098.stdout:Total 11 MB/s | 100 MB 00:08 2024-03-12T03:19:59.020 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction check 2024-03-12T03:19:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction check succeeded. 2024-03-12T03:19:59.387 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction test 2024-03-12T03:20:00.740 INFO:teuthology.orchestra.run.smithi098.stdout:Transaction test succeeded. 2024-03-12T03:20:00.741 INFO:teuthology.orchestra.run.smithi098.stdout:Running transaction 2024-03-12T03:20:03.659 INFO:teuthology.orchestra.run.smithi098.stdout: Preparing : 1/1 2024-03-12T03:20:04.309 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:20:04.548 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:20:07.365 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:20:07.575 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:20:10.424 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:10.424 INFO:teuthology.orchestra.run.smithi153.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:20:10.424 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:20:10.506 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:10.965 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:10.965 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:20:10.965 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:20:10.965 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:20:11.394 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-03-12T03:20:11.918 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:20:12.057 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:20:12.058 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:20:12.058 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:20:12.058 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:20:12.058 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:20:12.058 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-12T03:20:12.058 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:20:12.059 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:20:12.180 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:20:12.180 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:20:12.180 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-03-12T03:20:12.215 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-03-12T03:20:12.215 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-03-12T03:20:12.215 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-03-12T03:20:12.215 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-03-12T03:20:12.216 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-03-12T03:20:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:20:12.217 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:20:12.217 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:20:12.217 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:20:12.217 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-03-12T03:20:12.217 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:20:12.217 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:20:12.218 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:20:12.218 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:20:12.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:20:12.312 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:20:12.445 INFO:teuthology.orchestra.run.smithi098.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:12.848 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-03-12T03:20:14.235 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:20:14.311 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-03-12T03:20:14.313 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-03-12T03:20:14.328 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.f5ga8jtKZ2 2024-03-12T03:20:14.328 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.f5ga8jtKZ2 2024-03-12T03:20:14.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.f5ga8jtKZ2 2024-03-12T03:20:14.540 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.f5ga8jtKZ2 is 6KB 2024-03-12T03:20:14.591 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.f5ga8jtKZ2 2024-03-12T03:20:14.605 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:20:14.670 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:20:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:20:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:20:14.671 INFO:teuthology.orchestra.run.smithi153.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:20:14.672 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:20:14.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-03-12T03:20:17.211 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:44.827 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-12T03:20:44.828 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-12T03:20:44.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:20:46.452 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:46.453 INFO:teuthology.orchestra.run.smithi098.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-12T03:20:46.453 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:20:46.537 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:46.997 INFO:teuthology.orchestra.run.smithi098.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:46.997 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-12T03:20:46.997 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-12T03:20:46.998 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-12T03:20:47.427 INFO:teuthology.orchestra.run.smithi098.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout:Installed: 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:20:47.428 INFO:teuthology.orchestra.run.smithi098.stdout:Complete! 2024-03-12T03:20:47.963 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:20:48.107 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:20:48.107 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:20:48.107 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-12T03:20:48.107 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-12T03:20:48.107 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-12T03:20:48.108 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-12T03:20:48.108 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi098.front.sepia.ceph.com, path=None, version=distro) 2024-03-12T03:20:48.108 DEBUG:teuthology.orchestra.run.smithi098:> rpm -q kernel | sort -rV | head -n 1 2024-03-12T03:20:48.229 INFO:teuthology.orchestra.run.smithi098.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-12T03:20:48.229 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-12T03:20:48.230 DEBUG:teuthology.orchestra.run.smithi098:> sudo rpm -qi grub2-tools 2024-03-12T03:20:48.262 INFO:teuthology.orchestra.run.smithi098.stdout:Name : grub2-tools 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Epoch : 1 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Version : 2.06 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Release : 68.el9 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Architecture: x86_64 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Group : Unspecified 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Size : 8264254 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:License : GPLv3+ 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-12T03:20:48.263 INFO:teuthology.orchestra.run.smithi098.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:Packager : builder@centos.org 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:Vendor : CentOS 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:Summary : Support tools for GRUB. 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:Description : 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout:hardware devices. 2024-03-12T03:20:48.264 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:20:48.265 INFO:teuthology.orchestra.run.smithi098.stdout:This subpackage provides tools for support of all platforms. 2024-03-12T03:20:48.265 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-12T03:20:48.265 INFO:teuthology.task.kernel:Updating grub on smithi098 to boot 5.14.0-427.el9.x86_64 2024-03-12T03:20:48.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-12T03:20:48.870 INFO:teuthology.orchestra.run.smithi098.stderr:Generating grub configuration file ... 2024-03-12T03:20:50.275 INFO:teuthology.orchestra.run.smithi098.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-12T03:20:50.308 INFO:teuthology.orchestra.run.smithi098.stderr:done 2024-03-12T03:20:50.311 DEBUG:teuthology.orchestra.run.smithi098:> mktemp 2024-03-12T03:20:50.325 INFO:teuthology.orchestra.run.smithi098.stdout:/tmp/tmp.eWiRCOQoyW 2024-03-12T03:20:50.326 DEBUG:teuthology.orchestra.run.smithi098:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.eWiRCOQoyW 2024-03-12T03:20:50.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0666 /tmp/tmp.eWiRCOQoyW 2024-03-12T03:20:50.534 DEBUG:teuthology.orchestra.remote:smithi098:/tmp/tmp.eWiRCOQoyW is 6KB 2024-03-12T03:20:50.582 DEBUG:teuthology.orchestra.run.smithi098:> rm -fr /tmp/tmp.eWiRCOQoyW 2024-03-12T03:20:50.596 DEBUG:teuthology.orchestra.run.smithi098:> sudo /bin/ls /boot/loader/entries || true 2024-03-12T03:20:50.661 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-12T03:20:50.661 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-12T03:20:50.661 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-12T03:20:50.661 INFO:teuthology.orchestra.run.smithi098.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-12T03:20:50.662 DEBUG:teuthology.orchestra.run.smithi098:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-12T03:20:50.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo shutdown -r now 2024-03-12T03:21:03.265 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-12T03:21:12.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-12T03:21:12.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:21:15.330 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-12T03:21:20.814 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-12T03:21:20.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-12T03:21:20.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:21:27.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-12T03:21:27.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:21:39.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-12T03:21:45.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-03-12T03:21:48.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-12T03:21:48.365 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:21:51.422 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-12T03:22:00.770 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-12T03:22:00.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:22:01.230 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-12T03:22:01.638 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-12T03:22:01.638 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:22:01.638 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-03-12T03:22:01.652 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:22:01.652 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:22:01.652 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:22:01.653 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:22:02.653 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-12T03:22:02.654 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:22:02.713 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-03-12T03:22:02.780 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:03.424 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-12T03:22:03.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:22:21.858 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.98 2024-03-12T03:22:36.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-12T03:22:36.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:22:37.284 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-12T03:22:37.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-12T03:22:37.712 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-12T03:22:37.713 DEBUG:teuthology.orchestra.run.smithi098:> uname -r 2024-03-12T03:22:37.727 INFO:teuthology.orchestra.run.smithi098.stdout:5.14.0-427.el9.x86_64 2024-03-12T03:22:37.727 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-12T03:22:37.727 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-12T03:22:37.727 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-12T03:22:38.728 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-12T03:22:38.728 DEBUG:teuthology.orchestra.run.smithi098:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-12T03:22:38.793 INFO:teuthology.orchestra.run.smithi098.stdout:ttyS1 2024-03-12T03:22:38.830 DEBUG:teuthology.parallel:result is None 2024-03-12T03:22:38.830 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-12T03:22:38.836 INFO:teuthology.task.internal:Creating test directory... 2024-03-12T03:22:38.837 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:22:38.839 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-12T03:22:38.858 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-12T03:22:38.884 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-12T03:22:38.925 INFO:teuthology.task.internal:Creating archive directory... 2024-03-12T03:22:38.926 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:22:38.928 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-12T03:22:38.955 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-12T03:22:38.992 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-12T03:22:38.992 DEBUG:teuthology.orchestra.run.smithi098:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:22:38.994 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-12T03:22:39.028 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:22:39.042 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:22:39.043 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:22:39.056 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-12T03:22:39.058 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-12T03:22:39.064 INFO:teuthology.task.internal:Configuring sudo... 2024-03-12T03:22:39.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:22:39.067 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-12T03:22:39.114 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-12T03:22:39.123 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-12T03:22:39.124 DEBUG:teuthology.orchestra.run.smithi098:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:22:39.131 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-12T03:22:39.170 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:22:39.223 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:22:39.307 DEBUG:teuthology.orchestra.run.smithi098:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:22:39.350 DEBUG:teuthology.orchestra.run.smithi098:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:22:39.388 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:22:39.388 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:22:39.452 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:22:39.497 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-12T03:22:39.566 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:22:39.610 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-12T03:22:39.654 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-12T03:22:39.655 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-12T03:22:39.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo service rsyslog restart 2024-03-12T03:22:39.726 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-03-12T03:22:39.769 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:22:39.825 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T03:22:40.244 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-12T03:22:40.253 INFO:teuthology.task.internal:Starting timer... 2024-03-12T03:22:40.253 INFO:teuthology.run_tasks:Running task pcp... 2024-03-12T03:22:40.304 INFO:teuthology.run_tasks:Running task selinux... 2024-03-12T03:22:40.347 DEBUG:teuthology.orchestra.run.smithi098:> sudo service auditd rotate 2024-03-12T03:22:40.421 INFO:teuthology.orchestra.run.smithi098.stdout:Rotating logs: 2024-03-12T03:22:40.423 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-03-12T03:22:40.487 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-03-12T03:22:40.489 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-12T03:22:40.489 DEBUG:teuthology.orchestra.run.smithi098:> /usr/sbin/getenforce 2024-03-12T03:22:40.543 INFO:teuthology.orchestra.run.smithi098.stdout:Permissive 2024-03-12T03:22:40.543 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-03-12T03:22:40.569 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-03-12T03:22:40.569 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi098.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi153.front.sepia.ceph.com': 'permissive'} 2024-03-12T03:22:40.569 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T03:22:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:22:40.597 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T03:22:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T03:22:40.637 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-12T03:22:40.638 DEBUG:teuthology.orchestra.run.smithi098:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:22:40.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-03-12T03:22:40.796 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-12T03:22:40.806 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-12T03:22:40.823 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-12T03:22:40.824 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi098.front.sepia.ceph.com,smithi153.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-12T03:29:01.779 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi098.front.sepia.ceph.com'), Remote(name='ubuntu@smithi153.front.sepia.ceph.com')] 2024-03-12T03:29:01.781 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-12T03:29:01.782 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi098.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:29:01.877 DEBUG:teuthology.orchestra.run.smithi098:> true 2024-03-12T03:29:01.957 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi098.front.sepia.ceph.com' 2024-03-12T03:29:01.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-12T03:29:01.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-12T03:29:02.050 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-03-12T03:29:02.191 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-03-12T03:29:02.191 INFO:teuthology.run_tasks:Running task clock... 2024-03-12T03:29:02.204 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-12T03:29:02.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:29:02.205 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:29:02.207 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-12T03:29:02.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T03:29:02.238 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:29:02.255 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:29:02.262 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-12T03:29:02.278 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-12T03:29:02.285 INFO:teuthology.orchestra.run.smithi098.stderr:sudo: ntpd: command not found 2024-03-12T03:29:02.298 INFO:teuthology.orchestra.run.smithi098.stdout:506 Cannot talk to daemon 2024-03-12T03:29:02.308 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-03-12T03:29:02.315 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:29:02.322 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-03-12T03:29:02.332 INFO:teuthology.orchestra.run.smithi098.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:29:02.338 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-12T03:29:02.355 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-12T03:29:02.385 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:29:02.389 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:29:02.389 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-12T03:29:02.389 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.389 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.389 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.389 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.406 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-12T03:29:02.410 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T03:29:02.410 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-12T03:29:02.410 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.410 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.410 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.410 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T03:29:02.411 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-12T03:29:02.501 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-12T03:29:02.501 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-12T03:29:02.502 INFO:tasks.cephadm:Cluster fsid is ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:02.502 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-12T03:29:02.502 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.98', 'mon.c': '[v2:172.21.15.98:3301,v1:172.21.15.98:6790]', 'mon.b': '172.21.15.153'} 2024-03-12T03:29:02.502 INFO:tasks.cephadm:First mon is mon.a on smithi098 2024-03-12T03:29:02.502 INFO:tasks.cephadm:First mgr is y 2024-03-12T03:29:02.502 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-12T03:29:02.503 DEBUG:teuthology.orchestra.run.smithi098:> sudo hostname $(hostname -s) 2024-03-12T03:29:02.531 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-03-12T03:29:02.559 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-12T03:29:02.560 DEBUG:teuthology.orchestra.run.smithi098:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:29:03.004 INFO:teuthology.orchestra.run.smithi098.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 12 03:29 /home/ubuntu/cephtest/cephadm 2024-03-12T03:29:03.004 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-12T03:29:03.250 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 12 03:29 /home/ubuntu/cephtest/cephadm 2024-03-12T03:29:03.251 DEBUG:teuthology.orchestra.run.smithi098:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:29:03.267 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-12T03:29:03.293 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-12T03:29:03.293 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-12T03:29:03.309 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-12T03:29:04.131 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-12T03:29:04.244 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout: "repo_digests": [ 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-12T03:29:22.752 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:29:25.072 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-03-12T03:29:25.072 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-12T03:29:25.072 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-12T03:29:25.073 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-03-12T03:29:25.073 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-12T03:29:25.073 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-12T03:29:25.073 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-03-12T03:29:25.073 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-03-12T03:29:25.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo mkdir -p /etc/ceph 2024-03-12T03:29:25.126 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-03-12T03:29:25.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 777 /etc/ceph 2024-03-12T03:29:25.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-03-12T03:29:25.223 INFO:tasks.cephadm:Writing seed config... 2024-03-12T03:29:25.224 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-12T03:29:25.225 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-12T03:29:25.226 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:29:25.226 DEBUG:teuthology.orchestra.run.smithi098:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-12T03:29:25.252 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ac0aaff8-e020-11ee-95c8-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-12T03:29:25.253 DEBUG:teuthology.orchestra.run.smithi098:mon.a> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service 2024-03-12T03:29:25.295 DEBUG:teuthology.orchestra.run.smithi098:mgr.y> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service 2024-03-12T03:29:25.338 INFO:tasks.cephadm:Bootstrapping... 2024-03-12T03:29:25.338 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.98 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:25.534 INFO:teuthology.orchestra.run.smithi098.stderr:-------------------------------------------------------------------------------- 2024-03-12T03:29:25.534 INFO:teuthology.orchestra.run.smithi098.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'ac0aaff8-e020-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.98', '--skip-admin-label'] 2024-03-12T03:29:25.556 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: 4.9.3-dev 2024-03-12T03:29:25.557 INFO:teuthology.orchestra.run.smithi098.stderr:Verifying podman|docker is present... 2024-03-12T03:29:25.578 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: 4.9.3-dev 2024-03-12T03:29:25.580 INFO:teuthology.orchestra.run.smithi098.stderr:Verifying lvm2 is present... 2024-03-12T03:29:25.580 INFO:teuthology.orchestra.run.smithi098.stderr:Verifying time synchronization is in place... 2024-03-12T03:29:25.587 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:29:25.593 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: inactive 2024-03-12T03:29:25.599 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: enabled 2024-03-12T03:29:25.605 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: active 2024-03-12T03:29:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:Unit chronyd.service is enabled and running 2024-03-12T03:29:25.606 INFO:teuthology.orchestra.run.smithi098.stderr:Repeating the final host check... 2024-03-12T03:29:25.627 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: 4.9.3-dev 2024-03-12T03:29:25.629 INFO:teuthology.orchestra.run.smithi098.stderr:podman (/bin/podman) version 4.9.3 is present 2024-03-12T03:29:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl is present 2024-03-12T03:29:25.630 INFO:teuthology.orchestra.run.smithi098.stderr:lvcreate is present 2024-03-12T03:29:25.635 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-12T03:29:25.641 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: inactive 2024-03-12T03:29:25.648 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: enabled 2024-03-12T03:29:25.654 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: active 2024-03-12T03:29:25.655 INFO:teuthology.orchestra.run.smithi098.stderr:Unit chronyd.service is enabled and running 2024-03-12T03:29:25.655 INFO:teuthology.orchestra.run.smithi098.stderr:Host looks OK 2024-03-12T03:29:25.655 INFO:teuthology.orchestra.run.smithi098.stderr:Cluster fsid: ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:25.655 INFO:teuthology.orchestra.run.smithi098.stderr:Acquiring lock 140316703702944 on /run/cephadm/ac0aaff8-e020-11ee-95c8-87774f69a715.lock 2024-03-12T03:29:25.655 INFO:teuthology.orchestra.run.smithi098.stderr:Lock 140316703702944 acquired on /run/cephadm/ac0aaff8-e020-11ee-95c8-87774f69a715.lock 2024-03-12T03:29:25.656 INFO:teuthology.orchestra.run.smithi098.stderr:Verifying IP 172.21.15.98 port 3300 ... 2024-03-12T03:29:25.656 INFO:teuthology.orchestra.run.smithi098.stderr:Verifying IP 172.21.15.98 port 6789 ... 2024-03-12T03:29:25.656 INFO:teuthology.orchestra.run.smithi098.stderr:Base mon IP is 172.21.15.98, final addrv is [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-03-12T03:29:25.658 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.98 metric 100 2024-03-12T03:29:25.658 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.98 metric 100 2024-03-12T03:29:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-12T03:29:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-12T03:29:25.661 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-03-12T03:29:25.663 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-12T03:29:25.664 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-12T03:29:25.664 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-12T03:29:25.664 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-12T03:29:25.664 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d26b/64 scope link 2024-03-12T03:29:25.664 INFO:teuthology.orchestra.run.smithi098.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-12T03:29:25.664 INFO:teuthology.orchestra.run.smithi098.stderr:Mon IP `172.21.15.98` is in CIDR network `172.21.0.0/20` 2024-03-12T03:29:25.665 INFO:teuthology.orchestra.run.smithi098.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-12T03:29:25.665 INFO:teuthology.orchestra.run.smithi098.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-12T03:29:25.764 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-12T03:29:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Getting image source signatures 2024-03-12T03:29:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-12T03:29:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-12T03:29:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-12T03:29:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-12T03:29:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-12T03:29:26.167 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-12T03:29:26.314 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: Writing manifest to image destination 2024-03-12T03:29:26.467 INFO:teuthology.orchestra.run.smithi098.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-12T03:29:26.866 INFO:teuthology.orchestra.run.smithi098.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-12T03:29:27.358 INFO:teuthology.orchestra.run.smithi098.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-12T03:29:27.358 INFO:teuthology.orchestra.run.smithi098.stderr:Extracting ceph user uid/gid from container image... 2024-03-12T03:29:27.741 INFO:teuthology.orchestra.run.smithi098.stderr:stat: 167 167 2024-03-12T03:29:28.239 INFO:teuthology.orchestra.run.smithi098.stderr:Creating initial keys... 2024-03-12T03:29:28.663 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph-authtool: AQAYzO9l+95QIhAAn2HeeeEqL0ToFIgyg0xViQ== 2024-03-12T03:29:29.503 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph-authtool: AQAZzO9lg8K1GBAAt1DLni+jUv8kWuk4a1v+nw== 2024-03-12T03:29:30.433 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph-authtool: AQAazO9lGh6uFBAAqmFxrUVrsR+hp2zS/Ngblw== 2024-03-12T03:29:31.040 INFO:teuthology.orchestra.run.smithi098.stderr:Creating initial monmap... 2024-03-12T03:29:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:29:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-12T03:29:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:31.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:29:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:monmaptool for a [v2:172.21.15.98:3300,v1:172.21.15.98:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-12T03:29:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:setting min_mon_release = octopus 2024-03-12T03:29:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/monmaptool: set fsid to ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-12T03:29:31.929 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:29:31.929 INFO:teuthology.orchestra.run.smithi098.stderr:Creating mon... 2024-03-12T03:29:32.801 INFO:teuthology.orchestra.run.smithi098.stderr:create mon.a on 2024-03-12T03:29:33.007 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-12T03:29:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715.target. 2024-03-12T03:29:33.158 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715.target. 2024-03-12T03:29:33.446 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Failed to reset failed state of unit ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-12T03:29:33.454 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715.target.wants/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@.service. 2024-03-12T03:29:34.105 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:34 smithi098 podman[24409]: 2024-03-12 03:29:34.03732409 +0000 UTC m=+0.306604315 container start f846aab4cf07884a34b8fde4bb63842e76cb2bbff6cebf13a9cf6c74ac94af58 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, io.openshift.expose-services=, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8) 2024-03-12T03:29:34.137 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: disabled 2024-03-12T03:29:34.143 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: inactive 2024-03-12T03:29:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:firewalld.service is not enabled 2024-03-12T03:29:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:Not possible to enable service . firewalld.service is not available 2024-03-12T03:29:34.144 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for mon to start... 2024-03-12T03:29:34.145 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for mon... 2024-03-12T03:29:34.361 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:34 smithi098 ceph-mon[24428]: mkfs ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:34.361 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:34 smithi098 ceph-mon[24428]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:29:34.361 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:34 smithi098 bash[24409]: f846aab4cf07884a34b8fde4bb63842e76cb2bbff6cebf13a9cf6c74ac94af58 2024-03-12T03:29:34.361 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:34 smithi098 systemd[1]: Started Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:29:34.636 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: cluster: 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: id: ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: services: 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.536592s) 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: data: 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: pgs: 2024-03-12T03:29:34.637 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:35.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:35 smithi098 ceph-mon[24428]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:29:35.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:35 smithi098 ceph-mon[24428]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} 2024-03-12T03:29:35.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:35 smithi098 ceph-mon[24428]: fsmap 2024-03-12T03:29:35.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:35 smithi098 ceph-mon[24428]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:29:35.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:35 smithi098 ceph-mon[24428]: mgrmap e1: no daemons active 2024-03-12T03:29:35.176 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:35 smithi098 ceph-mon[24428]: from='client.? 172.21.15.98:0/2028353746' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:29:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:mon is available 2024-03-12T03:29:35.178 INFO:teuthology.orchestra.run.smithi098.stderr:Assimilating anything we can from ceph.conf... 2024-03-12T03:29:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:35.642 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: [global] 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: fsid = ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.98:3300,v1:172.21.15.98:6789] 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: [mgr] 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: [osd] 2024-03-12T03:29:35.643 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-12T03:29:35.644 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-12T03:29:36.143 INFO:teuthology.orchestra.run.smithi098.stderr:Generating new minimal ceph.conf... 2024-03-12T03:29:36.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:36 smithi098 ceph-mon[24428]: from='client.? 172.21.15.98:0/1113801183' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:29:36.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:36 smithi098 ceph-mon[24428]: from='client.? 172.21.15.98:0/1113801183' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T03:29:37.116 INFO:teuthology.orchestra.run.smithi098.stderr:Restarting the monitor... 2024-03-12T03:29:37.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 ceph-mon[24428]: from='client.? 172.21.15.98:0/841826785' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:29:37.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 systemd[1]: Stopping Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:29:37.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 bash[24646]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon.a" found: no such container 2024-03-12T03:29:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a[24424]: 2024-03-12T03:29:37.247+0000 7fdfa6f31700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:29:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a[24424]: 2024-03-12T03:29:37.247+0000 7fdfa6f31700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-12T03:29:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 podman[24659]: 2024-03-12 03:29:37.268834678 +0000 UTC m=+0.099604634 container died f846aab4cf07884a34b8fde4bb63842e76cb2bbff6cebf13a9cf6c74ac94af58 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, version=8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vendor=Red Hat, Inc., vcs-type=git, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_BRANCH=HEAD, name=centos-stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container) 2024-03-12T03:29:37.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 podman[24659]: 2024-03-12 03:29:37.623156431 +0000 UTC m=+0.453926396 container cleanup f846aab4cf07884a34b8fde4bb63842e76cb2bbff6cebf13a9cf6c74ac94af58 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, version=8, vcs-type=git, io.k8s.display-name=CentOS Stream 8, architecture=x86_64) 2024-03-12T03:29:37.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 bash[24659]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:29:37.892 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 bash[24685]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon.a" found: no such container 2024-03-12T03:29:38.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 podman[24673]: 2024-03-12 03:29:37.890838034 +0000 UTC m=+0.617274678 container remove f846aab4cf07884a34b8fde4bb63842e76cb2bbff6cebf13a9cf6c74ac94af58 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.openshift.expose-services=, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, release=754, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream) 2024-03-12T03:29:38.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T03:29:38.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 systemd[1]: Stopped Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:29:38.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:37 smithi098 systemd[1]: Starting Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:29:38.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 podman[24754]: 2024-03-12 03:29:38.018536301 +0000 UTC m=+0.018079225 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-12T03:29:38.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 podman[24754]: 2024-03-12 03:29:38.116528089 +0000 UTC m=+0.116071012 container create 6c18b4840264544c5e2ddba53391c208e5af879c87c6d72e4e608288272d060d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, release=754, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, io.openshift.expose-services=, architecture=x86_64, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, distribution-scope=public) 2024-03-12T03:29:38.373 INFO:teuthology.orchestra.run.smithi098.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 podman[24754]: 2024-03-12 03:29:38.30222089 +0000 UTC m=+0.301763831 container init 6c18b4840264544c5e2ddba53391c208e5af879c87c6d72e4e608288272d060d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, ceph=True, architecture=x86_64, io.openshift.expose-services=, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, version=8) 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 podman[24754]: 2024-03-12 03:29:38.305124879 +0000 UTC m=+0.304667803 container start 6c18b4840264544c5e2ddba53391c208e5af879c87c6d72e4e608288272d060d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, maintainer=Guillaume Abrioux , io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, version=8, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: pidfile_write: ignore empty --pid-file 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: load: jerasure load: lrc 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: RocksDB version: 6.15.5 2024-03-12T03:29:38.489 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Compile date Apr 18 2022 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: DB SUMMARY 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: DB Session ID: YDMM2P56HCGBACGWH2OH 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82633 ; 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:29:38.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.env: 0x5648e26c6860 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.fs: Posix File System 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.info_log: 0x5648e40cfdc0 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.statistics: (nil) 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.use_fsync: 0 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:29:38.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.db_log_dir: 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:29:38.492 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.write_buffer_manager: 0x5648e41c02a0 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.unordered_write: 0 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:29:38.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.row_cache: None 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.wal_filter: None 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.preserve_deletes: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:29:38.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:29:38.495 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:29:38.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_open_files: -1 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Compression algorithms supported: 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kZSTD supported: 0 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:29:38.497 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.merge_operator: 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_filter: None 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:29:38.498 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5648e409bd00) 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: hash_index_allow_collision: 1 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 1 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x5648e4106170 2024-03-12T03:29:38.499 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-03-12T03:29:38.500 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 4 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression: NoCompression 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:29:38.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.num_levels: 7 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:29:38.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:29:38.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.arena_block_size: 4194304 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:29:38.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:29:38.505 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.ttl: 2592000 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:29:38.506 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214178330413, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214178331103, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79183, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77446, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71755, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710214178, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "cde4c51e-7bca-4777-9103-737a42b044e0", "db_session_id": "YDMM2P56HCGBACGWH2OH"}} 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214178331432, "job": 1, "event": "recovery_finished"} 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5648e40eca80 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: DB pointer 0x5648e40fc000 2024-03-12T03:29:38.507 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.000 0 0 2024-03-12T03:29:38.508 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.000 0 0 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.000 0 0 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.5 0.00 0.00 1 0.000 0 0 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:29:38.509 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 26.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 26.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.000 0 0 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 163.5 0.00 0.00 1 0.000 0 0 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:29:38.510 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 163.5 0.00 0.00 1 0.000 0 0 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 26.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:29:38.511 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???) e1 preinit fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???).mds e1 new map 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???).mds e1 print_map 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:29:38.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expand map: {default=false} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta from 'false' to 'false' 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expanded map: {default=false} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expand map: {default=info} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta from 'info' to 'info' 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expanded map: {default=info} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expand map: {default=daemon} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta from 'daemon' to 'daemon' 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expanded map: {default=daemon} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expand map: {default=debug} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta from 'debug' to 'debug' 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: expand_channel_meta expanded map: {default=debug} 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 bash[24754]: 6c18b4840264544c5e2ddba53391c208e5af879c87c6d72e4e608288272d060d 2024-03-12T03:29:38.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 systemd[1]: Started Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:29:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-12T03:29:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: monmap e1: 1 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0]} 2024-03-12T03:29:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: fsmap 2024-03-12T03:29:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: osdmap e1: 0 total, 0 up, 0 in 2024-03-12T03:29:38.514 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:38 smithi098 ceph-mon[24773]: mgrmap e1: no daemons active 2024-03-12T03:29:39.430 INFO:teuthology.orchestra.run.smithi098.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-12T03:29:39.430 INFO:teuthology.orchestra.run.smithi098.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:29:39.430 INFO:teuthology.orchestra.run.smithi098.stderr:Creating mgr... 2024-03-12T03:29:39.431 INFO:teuthology.orchestra.run.smithi098.stderr:Verifying port 9283 ... 2024-03-12T03:29:39.597 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Failed to reset failed state of unit ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-12T03:29:39.604 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: Created symlink /etc/systemd/system/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715.target.wants/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@.service. 2024-03-12T03:29:40.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:39 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3464557643' entity='client.admin' 2024-03-12T03:29:40.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:39 smithi098 podman[24990]: 2024-03-12 03:29:39.887095526 +0000 UTC m=+0.017868103 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-12T03:29:40.244 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: disabled 2024-03-12T03:29:40.252 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: inactive 2024-03-12T03:29:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:firewalld.service is not enabled 2024-03-12T03:29:40.253 INFO:teuthology.orchestra.run.smithi098.stderr:Not possible to enable service . firewalld.service is not available 2024-03-12T03:29:40.261 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: disabled 2024-03-12T03:29:40.267 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: inactive 2024-03-12T03:29:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:firewalld.service is not enabled 2024-03-12T03:29:40.268 INFO:teuthology.orchestra.run.smithi098.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-12T03:29:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for mgr to start... 2024-03-12T03:29:40.269 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for mgr... 2024-03-12T03:29:40.384 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:40 smithi098 podman[24990]: 2024-03-12 03:29:40.162888598 +0000 UTC m=+0.293661170 container init 3c66c3a9135ba656713d1d24064af4437be41c7111ac7df6289ffa8277d0b847 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, vcs-type=git, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., name=centos-stream, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-12T03:29:40.384 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:40 smithi098 podman[24990]: 2024-03-12 03:29:40.165667297 +0000 UTC m=+0.296439868 container start 3c66c3a9135ba656713d1d24064af4437be41c7111ac7df6289ffa8277d0b847 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.component=centos-stream-container, version=8, io.openshift.expose-services=) 2024-03-12T03:29:40.384 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:40 smithi098 bash[24990]: 3c66c3a9135ba656713d1d24064af4437be41c7111ac7df6289ffa8277d0b847 2024-03-12T03:29:40.384 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:40 smithi098 systemd[1]: Started Ceph mgr.y for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:29:40.768 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: { 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "fsid": "ac0aaff8-e020-11ee-95c8-87774f69a715", 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "health": { 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "checks": {}, 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mutes": [] 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:40.769 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum": [ 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 0 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "a" 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "monmap": { 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:40.770 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osdmap": { 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-12T03:29:40.771 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-12T03:29:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-12T03:29:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-12T03:29:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "pgmap": { 2024-03-12T03:29:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-12T03:29:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-12T03:29:40.775 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "fsmap": { 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-12T03:29:40.776 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "available": false, 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "modules": [ 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "iostat", 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "nfs", 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "restful" 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "services": {} 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "servicemap": { 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "modified": "2024-03-12T03:29:34.097875+0000", 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "services": {} 2024-03-12T03:29:40.777 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-12T03:29:40.778 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: } 2024-03-12T03:29:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:40 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3771966433' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:29:41.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:40.656+0000 7f50e33e2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:29:41.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:40.757+0000 7f50e33e2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:29:41.375 INFO:teuthology.orchestra.run.smithi098.stderr:mgr not available, waiting (1/15)... 2024-03-12T03:29:41.680 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:41.357+0000 7f50e33e2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:29:41.680 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:41.448+0000 7f50e33e2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:29:41.680 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:41.533+0000 7f50e33e2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:29:41.943 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:41.677+0000 7f50e33e2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:29:41.943 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:41.755+0000 7f50e33e2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:29:41.943 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:41.860+0000 7f50e33e2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:29:41.944 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:41.940+0000 7f50e33e2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:29:42.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:42 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:42.099+0000 7f50e33e2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:29:42.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:42 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:42.181+0000 7f50e33e2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:29:43.500 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:43.196+0000 7f50e33e2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:29:43.500 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:43.324+0000 7f50e33e2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:29:43.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:43.524+0000 7f50e33e2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:29:43.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:43.608+0000 7f50e33e2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:29:43.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:43.740+0000 7f50e33e2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:29:44.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:44.050+0000 7f50e33e2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:29:44.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:44.265+0000 7f50e33e2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:29:44.647 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:44.348+0000 7f50e33e2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:29:44.647 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:44.515+0000 7f50e33e2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:29:45.707 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:45.413+0000 7f50e33e2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:29:46.012 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:45.704+0000 7f50e33e2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:29:46.012 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:45.797+0000 7f50e33e2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:29:46.012 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:45.944+0000 7f50e33e2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: Activating manager daemon y 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: mgrmap e2: y(active, starting, since 0.00288071s) 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: Manager daemon y is now available 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:29:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:29:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' 2024-03-12T03:29:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' 2024-03-12T03:29:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:46 smithi098 ceph-mon[24773]: from='mgr.14100 172.21.15.98:0/1126553860' entity='mgr.y' 2024-03-12T03:29:47.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:47 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3748226356' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:29:47.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:47 smithi098 ceph-mon[24773]: mgrmap e3: y(active, since 1.00613s) 2024-03-12T03:29:47.579 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: { 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "fsid": "ac0aaff8-e020-11ee-95c8-87774f69a715", 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "health": { 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "checks": {}, 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mutes": [] 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum": [ 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 0 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:47.580 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "a" 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "monmap": { 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osdmap": { 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-12T03:29:47.581 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "pgmap": { 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-12T03:29:47.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "fsmap": { 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "available": false, 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "modules": [ 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "iostat", 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "nfs", 2024-03-12T03:29:47.583 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "restful" 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "services": {} 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "servicemap": { 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "modified": "2024-03-12T03:29:34.097875+0000", 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "services": {} 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-12T03:29:47.584 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: } 2024-03-12T03:29:49.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:49 smithi098 ceph-mon[24773]: mgrmap e4: y(active, since 2s) 2024-03-12T03:29:53.368 INFO:teuthology.orchestra.run.smithi098.stderr:mgr not available, waiting (2/15)... 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: { 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "fsid": "ac0aaff8-e020-11ee-95c8-87774f69a715", 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "health": { 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "checks": {}, 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mutes": [] 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum": [ 2024-03-12T03:29:56.034 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 0 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "a" 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "quorum_age": 17, 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "monmap": { 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osdmap": { 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-12T03:29:56.035 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "pgmap": { 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-12T03:29:56.036 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "fsmap": { 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "available": true, 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "modules": [ 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "iostat", 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "nfs", 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "restful" 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ], 2024-03-12T03:29:56.037 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "services": {} 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "servicemap": { 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "modified": "2024-03-12T03:29:34.097875+0000", 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "services": {} 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: }, 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-12T03:29:56.038 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: } 2024-03-12T03:29:56.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:56 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2435062422' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-12T03:29:56.546 INFO:teuthology.orchestra.run.smithi098.stderr:mgr is available 2024-03-12T03:29:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:57.121 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: [global] 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: fsid = ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: [mgr] 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: [osd] 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-12T03:29:57.122 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-12T03:29:57.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:57 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1181194321' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-12T03:29:57.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:57 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1181194321' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-12T03:29:57.669 INFO:teuthology.orchestra.run.smithi098.stderr:Enabling cephadm module... 2024-03-12T03:29:58.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:58 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/241671071' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-12T03:29:58.773 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:58 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setuser ceph since I am not root 2024-03-12T03:29:58.773 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:58 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setgroup ceph since I am not root 2024-03-12T03:29:59.027 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:58 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:58.835+0000 7f898bc32000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:29:59.027 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:58 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:58.924+0000 7f898bc32000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:29:59.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: { 2024-03-12T03:29:59.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-12T03:29:59.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "available": true, 2024-03-12T03:29:59.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-12T03:29:59.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-12T03:29:59.440 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: } 2024-03-12T03:29:59.622 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:59.531+0000 7f898bc32000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:29:59.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:59 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/241671071' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-12T03:29:59.623 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:29:59 smithi098 ceph-mon[24773]: mgrmap e5: y(active, since 12s) 2024-03-12T03:29:59.936 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:59.620+0000 7f898bc32000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:29:59.937 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:59.704+0000 7f898bc32000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:29:59.937 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:59.852+0000 7f898bc32000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:29:59.937 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:29:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:29:59.933+0000 7f898bc32000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:29:59.940 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for the mgr to restart... 2024-03-12T03:29:59.940 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for mgr epoch 5... 2024-03-12T03:30:00.188 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:00.046+0000 7f898bc32000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:30:00.188 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:00.133+0000 7f898bc32000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:30:00.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:00 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/402908645' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:30:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:00.304+0000 7f898bc32000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:30:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:00.395+0000 7f898bc32000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:30:01.570 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:01.439+0000 7f898bc32000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:30:01.570 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:01.567+0000 7f898bc32000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:30:01.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:01.758+0000 7f898bc32000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:30:02.297 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:01.845+0000 7f898bc32000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:30:02.297 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:01.981+0000 7f898bc32000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:30:02.297 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:02.294+0000 7f898bc32000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:30:02.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:02.512+0000 7f898bc32000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:30:03.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:02.595+0000 7f898bc32000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:30:03.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:02.764+0000 7f898bc32000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:30:03.980 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:03.685+0000 7f898bc32000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:30:03.980 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:03.978+0000 7f898bc32000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:30:04.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:04.070+0000 7f898bc32000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:30:04.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:04.219+0000 7f898bc32000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:30:04.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:04 smithi098 ceph-mon[24773]: Active manager daemon y restarted 2024-03-12T03:30:04.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:04 smithi098 ceph-mon[24773]: Activating manager daemon y 2024-03-12T03:30:04.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:04 smithi098 ceph-mon[24773]: osdmap e2: 0 total, 0 up, 0 in 2024-03-12T03:30:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: { 2024-03-12T03:30:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-12T03:30:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "initialized": true 2024-03-12T03:30:05.358 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: } 2024-03-12T03:30:05.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: mgrmap e6: y(active, starting, since 0.110745s) 2024-03-12T03:30:05.652 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: Manager daemon y is now available 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:30:05.653 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:05 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:30:05.905 INFO:teuthology.orchestra.run.smithi098.stderr:mgr epoch 5 is available 2024-03-12T03:30:05.906 INFO:teuthology.orchestra.run.smithi098.stderr:Setting orchestrator backend to cephadm... 2024-03-12T03:30:06.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:06 smithi098 ceph-mon[24773]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:30:06.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:06 smithi098 ceph-mon[24773]: mgrmap e7: y(active, since 1.11539s) 2024-03-12T03:30:06.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:06 smithi098 ceph-mon[24773]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:30:06.809 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:06 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:30:06] ENGINE Bus STARTING 2024-03-12T03:30:06.810 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:06 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:30:06] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:30:06.810 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:06 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:30:06] ENGINE Bus STARTED 2024-03-12T03:30:07.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:07 smithi098 ceph-mon[24773]: mgrmap e8: y(active, since 2s) 2024-03-12T03:30:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:07 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:07 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:07 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:07 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:07.582 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: value unchanged 2024-03-12T03:30:08.127 INFO:teuthology.orchestra.run.smithi098.stderr:Generating ssh key... 2024-03-12T03:30:08.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:08 smithi098 ceph-mon[24773]: [12/Mar/2024:03:30:06] ENGINE Bus STARTING 2024-03-12T03:30:08.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:08 smithi098 ceph-mon[24773]: [12/Mar/2024:03:30:06] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:30:08.444 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:08 smithi098 ceph-mon[24773]: [12/Mar/2024:03:30:06] ENGINE Bus STARTED 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-mon[24773]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: Generating public/private rsa key pair. 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: Your identification has been saved in /tmp/tmpgqe42n68/key. 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: Your public key has been saved in /tmp/tmpgqe42n68/key.pub. 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: The key fingerprint is: 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: SHA256:wl45z2bqpDIgxDVGQvusFhQ/dBKzYjSFHUKFaUaGebo ceph-ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: The key's randomart image is: 2024-03-12T03:30:09.572 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: +---[RSA 3072]----+ 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: |=%OXo. | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: |=*X== | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: |+Bo+. | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: |+o+ .. . | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: |.o o o S | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: |E + . o + | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: | + . . . = | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: |. o o + | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: | o..o | 2024-03-12T03:30:09.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: +----[SHA256]-----+ 2024-03-12T03:30:10.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:10 smithi098 ceph-mon[24773]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:10.438 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:10 smithi098 ceph-mon[24773]: Generating ssh key... 2024-03-12T03:30:10.438 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGx3mhYAJpg78UYLQ/Q+yad5d0Cd8VHNxwVuV5Bd3zK5xrZkDtL4Qfj5oAvvZz/uUDsCY+EHVLq5j492/bZQWoPRBqOCNqbJp1xHh/SbsKU0ws+ECREv1OuBf/xT0yIv/sXYPWwL4wHRGEMrLpKWsTgKaluZqVcWJGXJg/XIbkSANwHPGIZBsn2JCIGYwCOY12+PDlUwdUSggNhxeG9GhY9QuI1ia18XvnxewIljBefilOl3eb332tZMb8qYFA960QJnz9bL75WDu4JZ2YAdigPrk+VpnWekqOP0JF53Hy6E9DSf/8Yh2aNGaqn0hxx8rXQGnySSCqsdniAG3Rzpej+H/KJNX6fyVOHw6R5F3E7T/FskfhbQI70RbTm7aSc0mNS/fchj82+MrbLi6c6/D1L/8KTzyyCFyM8UHuKRui2ffypzmXDw5NvRNrsQ0kscdcOu1itZRmwgdM920u6FG6RGKEClH6i+RkbuSrvAjuJ/g8PmI9oKV+lhwbRDWHuVc= ceph-ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:30:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-12T03:30:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:Adding key to root@localhost authorized_keys... 2024-03-12T03:30:11.015 INFO:teuthology.orchestra.run.smithi098.stderr:Adding host smithi098... 2024-03-12T03:30:11.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:11 smithi098 ceph-mon[24773]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:12.588 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: Added host 'smithi098' with addr '172.21.15.98' 2024-03-12T03:30:12.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:12 smithi098 ceph-mon[24773]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi098", "addr": "172.21.15.98", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:12.600 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:12 smithi098 ceph-mon[24773]: Deploying cephadm binary to smithi098 2024-03-12T03:30:13.189 INFO:teuthology.orchestra.run.smithi098.stderr:Deploying unmanaged mon service... 2024-03-12T03:30:13.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:13 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:13.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:13 smithi098 ceph-mon[24773]: Added host smithi098 2024-03-12T03:30:13.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:13 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:13.942 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-12T03:30:14.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:14 smithi098 ceph-mon[24773]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:14.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:14 smithi098 ceph-mon[24773]: Saving service mon spec with placement count:5 2024-03-12T03:30:14.991 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:14 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:15.413 INFO:teuthology.orchestra.run.smithi098.stderr:Deploying unmanaged mgr service... 2024-03-12T03:30:16.041 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-12T03:30:16.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:16 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:16 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:17.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:17 smithi098 ceph-mon[24773]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:17.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:17 smithi098 ceph-mon[24773]: Saving service mgr spec with placement count:2 2024-03-12T03:30:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:17 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:18.648 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:18 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1254115747' entity='client.admin' 2024-03-12T03:30:18.649 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:18 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:19.195 INFO:teuthology.orchestra.run.smithi098.stderr:Enabling the dashboard module... 2024-03-12T03:30:19.624 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:19 smithi098 ceph-mon[24773]: from='mgr.14118 172.21.15.98:0/2868654436' entity='mgr.y' 2024-03-12T03:30:19.624 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:19 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/63646531' entity='client.admin' 2024-03-12T03:30:20.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:20 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3761124525' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-12T03:30:20.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setuser ceph since I am not root 2024-03-12T03:30:20.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setgroup ceph since I am not root 2024-03-12T03:30:20.974 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:20.880+0000 7fb934c2e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:30:20.974 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:20.971+0000 7fb934c2e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:30:21.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:21 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3761124525' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-12T03:30:21.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:21 smithi098 ceph-mon[24773]: mgrmap e9: y(active, since 16s) 2024-03-12T03:30:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: { 2024-03-12T03:30:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-12T03:30:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "available": true, 2024-03-12T03:30:21.780 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-12T03:30:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-12T03:30:21.781 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: } 2024-03-12T03:30:21.945 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:21.619+0000 7fb934c2e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:30:21.945 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:21.709+0000 7fb934c2e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:30:21.945 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:21.794+0000 7fb934c2e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:30:21.946 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:21.943+0000 7fb934c2e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:30:22.220 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:22 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:22.024+0000 7fb934c2e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:30:22.220 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:22 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:22.133+0000 7fb934c2e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:30:22.220 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:22 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:22.217+0000 7fb934c2e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:30:22.348 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for the mgr to restart... 2024-03-12T03:30:22.348 INFO:teuthology.orchestra.run.smithi098.stderr:Waiting for mgr epoch 9... 2024-03-12T03:30:22.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:22 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2659909504' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-12T03:30:22.472 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:22 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:22.382+0000 7fb934c2e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:30:22.472 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:22 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:22.468+0000 7fb934c2e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:30:23.816 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:23.495+0000 7fb934c2e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:30:23.816 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:23.623+0000 7fb934c2e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:30:23.816 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:23.813+0000 7fb934c2e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:30:24.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:23.900+0000 7fb934c2e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:30:24.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:24.035+0000 7fb934c2e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:30:24.654 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:24.351+0000 7fb934c2e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:30:24.654 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:24.569+0000 7fb934c2e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:30:24.654 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:24.651+0000 7fb934c2e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:30:25.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:24.818+0000 7fb934c2e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:30:26.021 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:25 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:25.726+0000 7fb934c2e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:30:26.022 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:26 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:26.019+0000 7fb934c2e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:30:26.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:26 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:26.113+0000 7fb934c2e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:30:26.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:26 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:30:26.261+0000 7fb934c2e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:30:26.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:26 smithi098 ceph-mon[24773]: Active manager daemon y restarted 2024-03-12T03:30:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:26 smithi098 ceph-mon[24773]: Activating manager daemon y 2024-03-12T03:30:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:26 smithi098 ceph-mon[24773]: osdmap e3: 0 total, 0 up, 0 in 2024-03-12T03:30:27.290 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:30:27] ENGINE Bus STARTING 2024-03-12T03:30:27.291 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:30:27] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:30:27.349 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: { 2024-03-12T03:30:27.350 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-12T03:30:27.350 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: "initialized": true 2024-03-12T03:30:27.350 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: } 2024-03-12T03:30:27.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:30:27] ENGINE Bus STARTED 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: mgrmap e10: y(active, starting, since 0.0553359s) 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: Manager daemon y is now available 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:30:27.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:30:27.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:28.200 INFO:teuthology.orchestra.run.smithi098.stderr:mgr epoch 9 is available 2024-03-12T03:30:28.200 INFO:teuthology.orchestra.run.smithi098.stderr:Generating a dashboard self-signed certificate... 2024-03-12T03:30:28.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:28 smithi098 ceph-mon[24773]: [12/Mar/2024:03:30:27] ENGINE Bus STARTING 2024-03-12T03:30:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:28 smithi098 ceph-mon[24773]: [12/Mar/2024:03:30:27] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:30:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:28 smithi098 ceph-mon[24773]: [12/Mar/2024:03:30:27] ENGINE Bus STARTED 2024-03-12T03:30:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:28 smithi098 ceph-mon[24773]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-12T03:30:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:28 smithi098 ceph-mon[24773]: mgrmap e11: y(active, since 1.06546s) 2024-03-12T03:30:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:28 smithi098 ceph-mon[24773]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-12T03:30:28.455 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:28 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:28.930 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-12T03:30:29.524 INFO:teuthology.orchestra.run.smithi098.stderr:Creating initial admin user... 2024-03-12T03:30:29.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:29 smithi098 ceph-mon[24773]: mgrmap e12: y(active, since 2s) 2024-03-12T03:30:29.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:29 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:29.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:29 smithi098 ceph-mon[24773]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:29.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:29 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:29.942 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:29 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:30.379 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$MZOtGpGK4eQ2PL3vvCuHyuCMzF2ZtvWJ4v3BTlMBBxMJj8/2xhbN6", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710214230, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-12T03:30:31.059 INFO:teuthology.orchestra.run.smithi098.stderr:Fetching dashboard port number... 2024-03-12T03:30:31.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:31 smithi098 ceph-mon[24773]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:31.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:31 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:31.713 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: 8443 2024-03-12T03:30:32.667 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:32 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3537771062' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-12T03:30:32.681 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: disabled 2024-03-12T03:30:32.689 INFO:teuthology.orchestra.run.smithi098.stderr:systemctl: inactive 2024-03-12T03:30:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:firewalld.service is not enabled 2024-03-12T03:30:32.690 INFO:teuthology.orchestra.run.smithi098.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-12T03:30:32.693 INFO:teuthology.orchestra.run.smithi098.stderr:Ceph Dashboard is now available at: 2024-03-12T03:30:32.693 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:32.693 INFO:teuthology.orchestra.run.smithi098.stderr: URL: https://smithi098.front.sepia.ceph.com:8443/ 2024-03-12T03:30:32.693 INFO:teuthology.orchestra.run.smithi098.stderr: User: admin 2024-03-12T03:30:32.693 INFO:teuthology.orchestra.run.smithi098.stderr: Password: mpk4qqgfgn 2024-03-12T03:30:32.693 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:32.694 INFO:teuthology.orchestra.run.smithi098.stderr:Enabling autotune for osd_memory_target 2024-03-12T03:30:33.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:33 smithi098 ceph-mon[24773]: mgrmap e13: y(active, since 6s) 2024-03-12T03:30:34.665 INFO:teuthology.orchestra.run.smithi098.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:You can access the Ceph CLI with: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: ceph telemetry on 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr:For more information see: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-12T03:30:35.292 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T03:30:35.293 INFO:teuthology.orchestra.run.smithi098.stderr:Bootstrap complete. 2024-03-12T03:30:35.330 INFO:tasks.cephadm:Fetching config... 2024-03-12T03:30:35.330 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:30:35.330 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-12T03:30:35.346 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-12T03:30:35.346 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:30:35.346 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-12T03:30:35.401 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-12T03:30:35.401 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:30:35.401 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-12T03:30:35.468 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-12T03:30:35.468 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:30:35.468 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-12T03:30:35.523 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-12T03:30:35.523 DEBUG:teuthology.orchestra.run.smithi098:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGx3mhYAJpg78UYLQ/Q+yad5d0Cd8VHNxwVuV5Bd3zK5xrZkDtL4Qfj5oAvvZz/uUDsCY+EHVLq5j492/bZQWoPRBqOCNqbJp1xHh/SbsKU0ws+ECREv1OuBf/xT0yIv/sXYPWwL4wHRGEMrLpKWsTgKaluZqVcWJGXJg/XIbkSANwHPGIZBsn2JCIGYwCOY12+PDlUwdUSggNhxeG9GhY9QuI1ia18XvnxewIljBefilOl3eb332tZMb8qYFA960QJnz9bL75WDu4JZ2YAdigPrk+VpnWekqOP0JF53Hy6E9DSf/8Yh2aNGaqn0hxx8rXQGnySSCqsdniAG3Rzpej+H/KJNX6fyVOHw6R5F3E7T/FskfhbQI70RbTm7aSc0mNS/fchj82+MrbLi6c6/D1L/8KTzyyCFyM8UHuKRui2ffypzmXDw5NvRNrsQ0kscdcOu1itZRmwgdM920u6FG6RGKEClH6i+RkbuSrvAjuJ/g8PmI9oKV+lhwbRDWHuVc= ceph-ac0aaff8-e020-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:30:35.602 INFO:teuthology.orchestra.run.smithi098.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGx3mhYAJpg78UYLQ/Q+yad5d0Cd8VHNxwVuV5Bd3zK5xrZkDtL4Qfj5oAvvZz/uUDsCY+EHVLq5j492/bZQWoPRBqOCNqbJp1xHh/SbsKU0ws+ECREv1OuBf/xT0yIv/sXYPWwL4wHRGEMrLpKWsTgKaluZqVcWJGXJg/XIbkSANwHPGIZBsn2JCIGYwCOY12+PDlUwdUSggNhxeG9GhY9QuI1ia18XvnxewIljBefilOl3eb332tZMb8qYFA960QJnz9bL75WDu4JZ2YAdigPrk+VpnWekqOP0JF53Hy6E9DSf/8Yh2aNGaqn0hxx8rXQGnySSCqsdniAG3Rzpej+H/KJNX6fyVOHw6R5F3E7T/FskfhbQI70RbTm7aSc0mNS/fchj82+MrbLi6c6/D1L/8KTzyyCFyM8UHuKRui2ffypzmXDw5NvRNrsQ0kscdcOu1itZRmwgdM920u6FG6RGKEClH6i+RkbuSrvAjuJ/g8PmI9oKV+lhwbRDWHuVc= ceph-ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:30:35.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGx3mhYAJpg78UYLQ/Q+yad5d0Cd8VHNxwVuV5Bd3zK5xrZkDtL4Qfj5oAvvZz/uUDsCY+EHVLq5j492/bZQWoPRBqOCNqbJp1xHh/SbsKU0ws+ECREv1OuBf/xT0yIv/sXYPWwL4wHRGEMrLpKWsTgKaluZqVcWJGXJg/XIbkSANwHPGIZBsn2JCIGYwCOY12+PDlUwdUSggNhxeG9GhY9QuI1ia18XvnxewIljBefilOl3eb332tZMb8qYFA960QJnz9bL75WDu4JZ2YAdigPrk+VpnWekqOP0JF53Hy6E9DSf/8Yh2aNGaqn0hxx8rXQGnySSCqsdniAG3Rzpej+H/KJNX6fyVOHw6R5F3E7T/FskfhbQI70RbTm7aSc0mNS/fchj82+MrbLi6c6/D1L/8KTzyyCFyM8UHuKRui2ffypzmXDw5NvRNrsQ0kscdcOu1itZRmwgdM920u6FG6RGKEClH6i+RkbuSrvAjuJ/g8PmI9oKV+lhwbRDWHuVc= ceph-ac0aaff8-e020-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-12T03:30:35.664 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDGx3mhYAJpg78UYLQ/Q+yad5d0Cd8VHNxwVuV5Bd3zK5xrZkDtL4Qfj5oAvvZz/uUDsCY+EHVLq5j492/bZQWoPRBqOCNqbJp1xHh/SbsKU0ws+ECREv1OuBf/xT0yIv/sXYPWwL4wHRGEMrLpKWsTgKaluZqVcWJGXJg/XIbkSANwHPGIZBsn2JCIGYwCOY12+PDlUwdUSggNhxeG9GhY9QuI1ia18XvnxewIljBefilOl3eb332tZMb8qYFA960QJnz9bL75WDu4JZ2YAdigPrk+VpnWekqOP0JF53Hy6E9DSf/8Yh2aNGaqn0hxx8rXQGnySSCqsdniAG3Rzpej+H/KJNX6fyVOHw6R5F3E7T/FskfhbQI70RbTm7aSc0mNS/fchj82+MrbLi6c6/D1L/8KTzyyCFyM8UHuKRui2ffypzmXDw5NvRNrsQ0kscdcOu1itZRmwgdM920u6FG6RGKEClH6i+RkbuSrvAjuJ/g8PmI9oKV+lhwbRDWHuVc= ceph-ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:30:35.679 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-12T03:30:35.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:35 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/941270444' entity='client.admin' 2024-03-12T03:30:39.320 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-12T03:30:39.320 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-12T03:30:39.858 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:39 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3003705438' entity='client.admin' 2024-03-12T03:30:40.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:40.764 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi098", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:40.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:40.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:40.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:41.995 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi153 2024-03-12T03:30:41.996 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-12T03:30:41.996 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.conf 2024-03-12T03:30:42.014 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-12T03:30:42.014 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:30:42.071 INFO:tasks.cephadm:Adding host smithi153 to orchestrator... 2024-03-12T03:30:42.071 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch host add smithi153 2024-03-12T03:30:42.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:42.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:42.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:42.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:42.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:42.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:30:42.662 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:42.663 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:43.643 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:43 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:30:44.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:44 smithi098 ceph-mon[24773]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:44 smithi098 ceph-mon[24773]: Deploying cephadm binary to smithi153 2024-03-12T03:30:44.903 INFO:teuthology.orchestra.run.smithi098.stdout:Added host 'smithi153' with addr '172.21.15.153' 2024-03-12T03:30:45.498 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-12T03:30:46.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:45 smithi098 ceph-mon[24773]: Added host smithi153 2024-03-12T03:30:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:47.235 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:30:47.235 INFO:teuthology.orchestra.run.smithi098.stdout:[{"addr": "172.21.15.98", "hostname": "smithi098", "labels": [], "status": ""}, {"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}] 2024-03-12T03:30:47.526 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:47 smithi098 ceph-mon[24773]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:47.526 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:47 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:47.776 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-12T03:30:47.776 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-12T03:30:48.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:48 smithi098 ceph-mon[24773]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:30:49.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:49 smithi098 ceph-mon[24773]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:49 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:49 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:49 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:30:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:49 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:50.359 INFO:teuthology.orchestra.run.smithi098.stderr:adjusted tunables profile to default 2024-03-12T03:30:50.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:50 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/194900255' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:30:50.934 INFO:tasks.cephadm:Adding mon.a on smithi098 2024-03-12T03:30:50.934 INFO:tasks.cephadm:Adding mon.c on smithi098 2024-03-12T03:30:50.934 INFO:tasks.cephadm:Adding mon.b on smithi153 2024-03-12T03:30:50.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b' 2024-03-12T03:30:51.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:51 smithi098 ceph-mon[24773]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:51 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/194900255' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:30:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:51 smithi098 ceph-mon[24773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:30:52.802 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mon update... 2024-03-12T03:30:53.532 DEBUG:teuthology.orchestra.run.smithi098:mon.c> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.c.service 2024-03-12T03:30:53.534 DEBUG:teuthology.orchestra.run.smithi153:mon.b> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.b.service 2024-03-12T03:30:53.536 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:30:53.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:30:53.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:54.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:54 smithi098 ceph-mon[24773]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:54.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:54 smithi098 ceph-mon[24773]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b;count:3 2024-03-12T03:30:54.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:54 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:30:54.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:54 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:30:54.656 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:54 smithi098 ceph-mon[24773]: Deploying daemon mon.c on smithi098 2024-03-12T03:30:55.257 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:30:55.257 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":1,"fsid":"ac0aaff8-e020-11ee-95c8-87774f69a715","modified":"2024-03-12T03:29:31.340666Z","created":"2024-03-12T03:29:31.340666Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-12T03:30:55.259 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 1 2024-03-12T03:30:55.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:55 smithi098 ceph-mon[24773]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:55.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:30:55 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/487950005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:30:56.228 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 podman[29454]: 2024-03-12 03:30:56.128415899 +0000 UTC m=+0.119064191 container create bb746519eeb151cd78a1bc0bf4ea8f145247a6de0cd3bd594c3199f05e644858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vcs-type=git, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.openshift.expose-services=, ceph=True) 2024-03-12T03:30:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 podman[29454]: 2024-03-12 03:30:56.305919554 +0000 UTC m=+0.296567857 container init bb746519eeb151cd78a1bc0bf4ea8f145247a6de0cd3bd594c3199f05e644858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, RELEASE=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, architecture=x86_64, distribution-scope=public, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-12T03:30:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 podman[29454]: 2024-03-12 03:30:56.309026873 +0000 UTC m=+0.299675166 container start bb746519eeb151cd78a1bc0bf4ea8f145247a6de0cd3bd594c3199f05e644858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, RELEASE=HEAD, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, ceph=True, architecture=x86_64, release=754, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux ) 2024-03-12T03:30:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:30:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-12T03:30:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: pidfile_write: ignore empty --pid-file 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: load: jerasure load: lrc 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: RocksDB version: 6.15.5 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Compile date Apr 18 2022 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: DB SUMMARY 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: DB Session ID: BP76UHQ4RQN4EVF9G4OJ 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:30:56.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.env: 0x556f90ec6860 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.fs: Posix File System 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.info_log: 0x556f92887dc0 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.statistics: (nil) 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.use_fsync: 0 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:30:56.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.db_log_dir: 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:30:56.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.write_buffer_manager: 0x556f929782d0 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:30:56.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.unordered_write: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.row_cache: None 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.wal_filter: None 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.preserve_deletes: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:30:56.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:30:56.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_open_files: -1 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:30:56.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Compression algorithms supported: 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kZSTD supported: 0 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.merge_operator: 2024-03-12T03:30:56.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_filter: None 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556f92853d00) 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_type: 0 2024-03-12T03:30:56.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_index_type: 0 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_shortening: 1 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: hash_index_allow_collision: 1 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: checksum: 1 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: no_block_cache: 0 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache: 0x556f928be170 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_options: 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: capacity : 536870912 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: num_shard_bits : 4 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: strict_capacity_limit : 0 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:30:56.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_compressed: (nil) 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: persistent_cache: (nil) 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size: 4096 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size_deviation: 10 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_restart_interval: 16 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_block_restart_interval: 1 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: metadata_block_size: 4096 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: partition_filters: 0 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: use_delta_encoding: 1 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: whole_key_filtering: 1 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: verify_compression: 0 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: format_version: 4 2024-03-12T03:30:56.586 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_index_compression: 1 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_align: 0 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression: NoCompression 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.num_levels: 7 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:30:56.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:30:56.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.arena_block_size: 4194304 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:30:56.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:30:56.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.ttl: 2592000 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:30:56.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214256335230, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214256335526, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710214256, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f5a2bf00-bde9-4c54-a32f-d016e0a3bf5c", "db_session_id": "BP76UHQ4RQN4EVF9G4OJ"}} 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214256335840, "job": 1, "event": "recovery_finished"} 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:30:56.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x556f928a4a80 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: DB pointer 0x556f928b4000 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** DB Stats ** 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:30:56.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:30:56.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T03:30:56.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:30:56.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: using public_addrv [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: starting mon.c rank -1 at public addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] at bind addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(???) e0 preinit fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).mds e1 new map 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout: e1 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout: legacy client fscid: -1 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout: No filesystems configured 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:30:56.597 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/194900255' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/194900255' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:30:56.598 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b;count:3 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: Deploying daemon mon.c on smithi098 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:30:56.599 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/487950005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:30:56.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-12T03:30:56.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 ceph-mon[29473]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-12T03:30:56.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 bash[29454]: bb746519eeb151cd78a1bc0bf4ea8f145247a6de0cd3bd594c3199f05e644858 2024-03-12T03:30:56.600 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:30:56 smithi098 systemd[1]: Started Ceph mon.c for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:30:56.834 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:30:56.834 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:31:01.732 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:31:01.732 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"ac0aaff8-e020-11ee-95c8-87774f69a715","modified":"2024-03-12T03:30:56.360049Z","created":"2024-03-12T03:29:31.340666Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T03:31:01.735 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-03-12T03:31:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: mon.a calling monitor election 2024-03-12T03:31:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: mon.c calling monitor election 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: fsmap 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: mgrmap e13: y(active, since 35s) 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: overall HEALTH_OK 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:01.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[24773]: Deploying daemon mon.b on smithi153 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: mon.a calling monitor election 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: mon.c calling monitor election 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: fsmap 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: mgrmap e13: y(active, since 35s) 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: overall HEALTH_OK 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:01.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:01 smithi098 ceph-mon[29473]: Deploying daemon mon.b on smithi153 2024-03-12T03:31:02.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:02 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/3742282189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:02.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:02 smithi098 ceph-mon[24773]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:02.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:02 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:02.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:02 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/3742282189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:02.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:02 smithi098 ceph-mon[29473]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:02.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:02 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:03.284 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:31:03.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:31:05.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:05 smithi153 systemd[1]: Starting Ceph mon.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:31:05.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:05 smithi098 ceph-mon[24773]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:05 smithi098 ceph-mon[29473]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.252 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:31:06.252 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":2,"fsid":"ac0aaff8-e020-11ee-95c8-87774f69a715","modified":"2024-03-12T03:30:56.360049Z","created":"2024-03-12T03:29:31.340666Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-12T03:31:06.254 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 2 2024-03-12T03:31:06.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 podman[25611]: 2024-03-12 03:31:06.131979957 +0000 UTC m=+0.112678569 container create 61c20497c94577252efe7976dbe6a98d11641b1af719337c1f16d6aa4ad08f90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, release=754, architecture=x86_64, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , name=centos-stream) 2024-03-12T03:31:06.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 podman[25611]: 2024-03-12 03:31:06.039681539 +0000 UTC m=+0.020380158 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-03-12T03:31:06.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 podman[25611]: 2024-03-12 03:31:06.359519813 +0000 UTC m=+0.340218437 container init 61c20497c94577252efe7976dbe6a98d11641b1af719337c1f16d6aa4ad08f90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, ceph=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, version=8, maintainer=Guillaume Abrioux ) 2024-03-12T03:31:06.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 podman[25611]: 2024-03-12 03:31:06.362946389 +0000 UTC m=+0.343645000 container start 61c20497c94577252efe7976dbe6a98d11641b1af719337c1f16d6aa4ad08f90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_CLEAN=True, ceph=True, vendor=Red Hat, Inc., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, version=8, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, name=centos-stream, architecture=x86_64) 2024-03-12T03:31:06.385 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:31:06.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-12T03:31:06.386 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pidfile_write: ignore empty --pid-file 2024-03-12T03:31:06.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: load: jerasure load: lrc 2024-03-12T03:31:06.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: RocksDB version: 6.15.5 2024-03-12T03:31:06.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Compile date Apr 18 2022 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: DB SUMMARY 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: DB Session ID: YOZ6ICO5ZCJJ5J13YOIA 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:31:06.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.env: 0x557b8cec6860 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.fs: Posix File System 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.info_log: 0x557b8eabfdc0 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.statistics: (nil) 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.use_fsync: 0 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:31:06.695 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.db_log_dir: 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:31:06.696 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.write_buffer_manager: 0x557b8ebb0240 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.unordered_write: 0 2024-03-12T03:31:06.697 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.row_cache: None 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.wal_filter: None 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.preserve_deletes: 0 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:31:06.698 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:31:06.699 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_open_files: -1 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:31:06.701 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Compression algorithms supported: 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kZSTD supported: 0 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:31:06.702 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.merge_operator: 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_filter: None 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557b8ea8bd00) 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_type: 0 2024-03-12T03:31:06.703 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_index_type: 0 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_shortening: 1 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: hash_index_allow_collision: 1 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: checksum: 1 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: no_block_cache: 0 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache: 0x557b8eaf6170 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_options: 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: capacity : 536870912 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: num_shard_bits : 4 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: strict_capacity_limit : 0 2024-03-12T03:31:06.704 INFO:journalctl@ceph.mon.b.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_compressed: (nil) 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: persistent_cache: (nil) 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size: 4096 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size_deviation: 10 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_restart_interval: 16 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_block_restart_interval: 1 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: metadata_block_size: 4096 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: partition_filters: 0 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: use_delta_encoding: 1 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: whole_key_filtering: 1 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: verify_compression: 0 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: format_version: 4 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_index_compression: 1 2024-03-12T03:31:06.705 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_align: 0 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression: NoCompression 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.num_levels: 7 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:31:06.706 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:31:06.707 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:31:06.708 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.arena_block_size: 4194304 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:31:06.709 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:31:06.710 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.ttl: 2592000 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-03-12T03:31:06.711 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214266389831, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214266390126, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1710214266, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6fc0310f-caf3-42ba-a28c-b452603ba4dc", "db_session_id": "YOZ6ICO5ZCJJ5J13YOIA"}} 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710214266390428, "job": 1, "event": "recovery_finished"} 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x557b8eadca80 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: DB pointer 0x557b8eaec000 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** DB Stats ** 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:31:06.712 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:31:06.713 INFO:journalctl@ceph.mon.b.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:31:06.714 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.7 0.00 0.00 1 0.000 0 0 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:31:06.715 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: using public_addr v2:172.21.15.153:0/0 -> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: starting mon.b rank -1 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(???) e0 preinit fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).mds e1 new map 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: e1 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:31:06.716 INFO:journalctl@ceph.mon.b.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout: legacy client fscid: -1 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout: No filesystems configured 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:31:06.717 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/194900255' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/194900255' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:06.718 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: Saving service mon spec with placement smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b;count:3 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: Deploying daemon mon.c on smithi098 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/487950005' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.a calling monitor election 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:06.719 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.c calling monitor election 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: monmap e2: 2 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: fsmap 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mgrmap e13: y(active, since 35s) 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: overall HEALTH_OK 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:06.720 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: Deploying daemon mon.b on smithi153 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/3742282189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/4184489104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 ceph-mon[25643]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 bash[25611]: 61c20497c94577252efe7976dbe6a98d11641b1af719337c1f16d6aa4ad08f90 2024-03-12T03:31:06.721 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:06 smithi153 systemd[1]: Started Ceph mon.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:31:06.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:06 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/4184489104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:06 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/4184489104' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:08.124 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-12T03:31:08.125 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-12T03:31:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: mon.a calling monitor election 2024-03-12T03:31:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: mon.c calling monitor election 2024-03-12T03:31:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: mon.b calling monitor election 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: fsmap 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: mgrmap e13: y(active, since 45s) 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: overall HEALTH_OK 2024-03-12T03:31:11.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: mon.a calling monitor election 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: mon.c calling monitor election 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:11.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: mon.b calling monitor election 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: fsmap 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: mgrmap e13: y(active, since 45s) 2024-03-12T03:31:11.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: overall HEALTH_OK 2024-03-12T03:31:11.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:11.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:11.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:11.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: mon.a calling monitor election 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: mon.c calling monitor election 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: mon.b calling monitor election 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:11.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: fsmap 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: osdmap e4: 0 total, 0 up, 0 in 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: mgrmap e13: y(active, since 45s) 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: overall HEALTH_OK 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:11.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:13.025 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-03-12T03:31:13.025 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":3,"fsid":"ac0aaff8-e020-11ee-95c8-87774f69a715","modified":"2024-03-12T03:31:06.409882Z","created":"2024-03-12T03:29:31.340666Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3300","nonce":0},{"type":"v1","addr":"172.21.15.98:6789","nonce":0}]},"addr":"172.21.15.98:6789/0","public_addr":"172.21.15.98:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:3301","nonce":0},{"type":"v1","addr":"172.21.15.98:6790","nonce":0}]},"addr":"172.21.15.98:6790/0","public_addr":"172.21.15.98:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-12T03:31:13.027 INFO:teuthology.orchestra.run.smithi153.stderr:dumped monmap epoch 3 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:13.591 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:13.591 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:13.591 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:13 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/1670504770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:13.613 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-12T03:31:13.613 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:13.638 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/1670504770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:13.639 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:13.640 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:13.640 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:13.640 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:13 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/1670504770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[24773]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[24773]: Reconfiguring daemon mon.a on smithi098 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[29473]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[29473]: Reconfiguring daemon mon.a on smithi098 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:14.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:14.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:14.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:14 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:14 smithi153 ceph-mon[25643]: Reconfiguring mon.a (unknown last config time)... 2024-03-12T03:31:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:14 smithi153 ceph-mon[25643]: Reconfiguring daemon mon.a on smithi098 2024-03-12T03:31:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:14 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:14 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:14.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:14 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:14.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:14 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:15.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:15 smithi098 ceph-mon[29473]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:31:15.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:15 smithi098 ceph-mon[29473]: Reconfiguring daemon mon.c on smithi098 2024-03-12T03:31:15.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:15 smithi098 ceph-mon[29473]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:15.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:15 smithi098 ceph-mon[24773]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:31:15.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:15 smithi098 ceph-mon[24773]: Reconfiguring daemon mon.c on smithi098 2024-03-12T03:31:15.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:15 smithi098 ceph-mon[24773]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:15.569 INFO:teuthology.orchestra.run.smithi098.stdout:# minimal ceph.conf for ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:31:15.570 INFO:teuthology.orchestra.run.smithi098.stdout:[global] 2024-03-12T03:31:15.570 INFO:teuthology.orchestra.run.smithi098.stdout: fsid = ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:31:15.570 INFO:teuthology.orchestra.run.smithi098.stdout: mon_host = [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] 2024-03-12T03:31:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:15 smithi153 ceph-mon[25643]: Reconfiguring mon.c (monmap changed)... 2024-03-12T03:31:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:15 smithi153 ceph-mon[25643]: Reconfiguring daemon mon.c on smithi098 2024-03-12T03:31:15.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:15 smithi153 ceph-mon[25643]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:16.196 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-12T03:31:16.197 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:31:16.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:31:16.265 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:31:16.265 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:31:16.333 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-12T03:31:16.333 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-03-12T03:31:16.367 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-12T03:31:16.368 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:31:16.437 INFO:tasks.cephadm:Adding mgr.y on smithi098 2024-03-12T03:31:16.438 INFO:tasks.cephadm:Adding mgr.x on smithi153 2024-03-12T03:31:16.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi098=y;smithi153=x' 2024-03-12T03:31:16.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:16 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/269446489' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:16.738 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:16 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/269446489' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:16.738 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:16 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/269446489' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:17.670 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:17 smithi153 ceph-mon[25643]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:17.670 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:17 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:17.670 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:17 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:17.671 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:17 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:17.671 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:17 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:17.671 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:17 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:17.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[29473]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[24773]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:31:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:31:17.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:17 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:18.315 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled mgr update... 2024-03-12T03:31:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:18 smithi153 ceph-mon[25643]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:31:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:18 smithi153 ceph-mon[25643]: Reconfiguring daemon mon.b on smithi153 2024-03-12T03:31:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:18 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:18.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:18 smithi098 ceph-mon[24773]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:31:18.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:18 smithi098 ceph-mon[24773]: Reconfiguring daemon mon.b on smithi153 2024-03-12T03:31:18.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:18 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:18.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:18 smithi098 ceph-mon[29473]: Reconfiguring mon.b (monmap changed)... 2024-03-12T03:31:18.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:18 smithi098 ceph-mon[29473]: Reconfiguring daemon mon.b on smithi153 2024-03-12T03:31:18.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:18 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:19.077 DEBUG:teuthology.orchestra.run.smithi153:mgr.x> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service 2024-03-12T03:31:19.079 INFO:tasks.cephadm:Deploying OSDs... 2024-03-12T03:31:19.079 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:31:19.079 DEBUG:teuthology.orchestra.run.smithi098:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:31:19.098 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:31:19.098 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_1 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-12 03:30:55.854696995 +0000 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-12 03:28:40.314624879 +0000 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-12 03:28:40.314624879 +0000 2024-03-12T03:31:19.155 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-12 03:28:40.314624879 +0000 2024-03-12T03:31:19.156 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:31:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-12T03:31:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-12T03:31:19.225 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000270121 s, 1.9 MB/s 2024-03-12T03:31:19.226 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:31:19.285 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_2 2024-03-12T03:31:19.342 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:31:19.342 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:19.342 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-12T03:31:19.342 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:19.342 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:19.342 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-12 03:30:55.854696995 +0000 2024-03-12T03:31:19.342 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-12 03:28:40.644620704 +0000 2024-03-12T03:31:19.343 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-12 03:28:40.644620704 +0000 2024-03-12T03:31:19.343 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-12 03:28:40.644620704 +0000 2024-03-12T03:31:19.343 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:31:19.410 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-12T03:31:19.410 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-12T03:31:19.410 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.00019181 s, 2.7 MB/s 2024-03-12T03:31:19.411 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:31:19.469 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_3 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-12 03:30:55.854696995 +0000 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-12 03:28:40.979616465 +0000 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-12 03:28:40.979616465 +0000 2024-03-12T03:31:19.525 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-12 03:28:40.979616465 +0000 2024-03-12T03:31:19.526 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:31:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:19 smithi098 ceph-mon[24773]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi098=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:19 smithi098 ceph-mon[24773]: Saving service mgr spec with placement smithi098=y;smithi153=x;count:2 2024-03-12T03:31:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:19 smithi098 ceph-mon[24773]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:19 smithi098 ceph-mon[29473]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi098=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:19 smithi098 ceph-mon[29473]: Saving service mgr spec with placement smithi098=y;smithi153=x;count:2 2024-03-12T03:31:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:19 smithi098 ceph-mon[29473]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:19.591 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-12T03:31:19.591 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-12T03:31:19.592 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000222481 s, 2.3 MB/s 2024-03-12T03:31:19.592 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:31:19.649 DEBUG:teuthology.orchestra.run.smithi098:> stat /dev/vg_nvme/lv_4 2024-03-12T03:31:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:19 smithi153 ceph-mon[25643]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi098=y;smithi153=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:19 smithi153 ceph-mon[25643]: Saving service mgr spec with placement smithi098=y;smithi153=x;count:2 2024-03-12T03:31:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:19 smithi153 ceph-mon[25643]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:31:19.704 INFO:teuthology.orchestra.run.smithi098.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:19.704 INFO:teuthology.orchestra.run.smithi098.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-12T03:31:19.704 INFO:teuthology.orchestra.run.smithi098.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:19.704 INFO:teuthology.orchestra.run.smithi098.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:19.704 INFO:teuthology.orchestra.run.smithi098.stdout:Access: 2024-03-12 03:30:55.855696982 +0000 2024-03-12T03:31:19.704 INFO:teuthology.orchestra.run.smithi098.stdout:Modify: 2024-03-12 03:28:41.316612200 +0000 2024-03-12T03:31:19.705 INFO:teuthology.orchestra.run.smithi098.stdout:Change: 2024-03-12 03:28:41.316612200 +0000 2024-03-12T03:31:19.705 INFO:teuthology.orchestra.run.smithi098.stdout: Birth: 2024-03-12 03:28:41.316612200 +0000 2024-03-12T03:31:19.705 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:31:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records in 2024-03-12T03:31:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:1+0 records out 2024-03-12T03:31:19.770 INFO:teuthology.orchestra.run.smithi098.stderr:512 bytes copied, 0.000204517 s, 2.5 MB/s 2024-03-12T03:31:19.772 DEBUG:teuthology.orchestra.run.smithi098:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:31:19.828 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-12T03:31:19.828 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-03-12T03:31:19.842 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-12T03:31:19.842 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-12 03:31:05.063527477 +0000 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-12 03:26:30.510947022 +0000 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-12 03:26:30.510947022 +0000 2024-03-12T03:31:19.898 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-12 03:26:30.510947022 +0000 2024-03-12T03:31:19.899 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-12T03:31:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-12T03:31:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-12T03:31:19.964 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000245806 s, 2.1 MB/s 2024-03-12T03:31:19.965 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-12T03:31:20.022 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-03-12T03:31:20.078 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-12T03:31:20.078 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:20.078 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-12T03:31:20.078 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:20.078 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:20.079 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-12 03:31:05.063527477 +0000 2024-03-12T03:31:20.079 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-12 03:26:30.872940433 +0000 2024-03-12T03:31:20.079 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-12 03:26:30.872940433 +0000 2024-03-12T03:31:20.079 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-12 03:26:30.872940433 +0000 2024-03-12T03:31:20.079 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-12T03:31:20.144 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-12T03:31:20.144 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-12T03:31:20.144 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000172291 s, 3.0 MB/s 2024-03-12T03:31:20.145 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-12T03:31:20.201 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-03-12T03:31:20.255 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-12T03:31:20.255 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:20.255 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-12T03:31:20.255 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:20.255 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:20.256 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-12 03:31:05.064527458 +0000 2024-03-12T03:31:20.256 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-12 03:26:31.254933480 +0000 2024-03-12T03:31:20.256 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-12 03:26:31.254933480 +0000 2024-03-12T03:31:20.256 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-12 03:26:31.254933480 +0000 2024-03-12T03:31:20.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-12T03:31:20.321 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-12T03:31:20.321 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-12T03:31:20.321 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000193286 s, 2.6 MB/s 2024-03-12T03:31:20.322 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-12T03:31:20.379 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-03-12T03:31:20.433 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-12T03:31:20.433 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-12T03:31:20.433 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-12T03:31:20.434 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-12T03:31:20.434 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-03-12T03:31:20.434 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-03-12 03:31:05.064527458 +0000 2024-03-12T03:31:20.434 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-03-12 03:26:31.591927346 +0000 2024-03-12T03:31:20.434 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-03-12 03:26:31.591927346 +0000 2024-03-12T03:31:20.434 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-03-12 03:26:31.591927346 +0000 2024-03-12T03:31:20.434 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-12T03:31:20.500 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-03-12T03:31:20.501 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-03-12T03:31:20.501 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000188256 s, 2.7 MB/s 2024-03-12T03:31:20.502 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-12T03:31:20.557 INFO:tasks.cephadm:Deploying osd.0 on smithi098 with /dev/vg_nvme/lv_4... 2024-03-12T03:31:20.557 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:31:20.871 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:20.871 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:20.871 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:20.871 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:20.871 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:20.871 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: Deploying daemon mgr.x on smithi153 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[29473]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:20.872 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:31:20.873 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:31:20.873 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:31:20.873 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:20.873 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: Deploying daemon mgr.x on smithi153 2024-03-12T03:31:20.873 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:20 smithi098 ceph-mon[24773]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:21.163 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: Deploying daemon mgr.x on smithi153 2024-03-12T03:31:21.164 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:20 smithi153 ceph-mon[25643]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:22.604 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:31:22.604 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:31:22.604 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-12T03:31:22.604 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-12T03:31:22.604 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01132 s, 926 MB/s 2024-03-12T03:31:22.604 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-12T03:31:23.158 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_4 2024-03-12T03:31:23.181 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.181 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:23.181 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:23.181 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:23.181 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:23.182 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:31:23.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:31:23.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:23.308 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.223+0000 7f9b1089d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:31:23.308 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.306+0000 7f9b1089d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:31:23.309 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:23.645 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.455+0000 7f9b1089d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:31:23.645 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.536+0000 7f9b1089d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:31:23.645 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.643+0000 7f9b1089d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:31:23.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.726+0000 7f9b1089d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:31:23.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.889+0000 7f9b1089d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:31:24.356 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:23.973+0000 7f9b1089d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[24773]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[24773]: Reconfiguring daemon mgr.y on smithi098 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[29473]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[29473]: Reconfiguring daemon mgr.y on smithi098 2024-03-12T03:31:24.516 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:24.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:24.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:24.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:24.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:24 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:24.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:24 smithi153 ceph-mon[25643]: Reconfiguring mgr.y (unknown last config time)... 2024-03-12T03:31:24.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:24 smithi153 ceph-mon[25643]: Reconfiguring daemon mgr.y on smithi098 2024-03-12T03:31:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:24 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:24 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:24 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:24 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:24 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:25.340 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:25.018+0000 7f9b1089d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:31:25.341 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:25.147+0000 7f9b1089d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:31:25.341 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:25.338+0000 7f9b1089d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:31:25.633 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:25 smithi098 ceph-mon[24773]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:25.633 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:25 smithi098 ceph-mon[29473]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:25 smithi153 ceph-mon[25643]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:25.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:25.426+0000 7f9b1089d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:31:25.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:25.559+0000 7f9b1089d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:31:26.181 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:25.879+0000 7f9b1089d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:31:26.182 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:26.096+0000 7f9b1089d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:31:26.182 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:26.179+0000 7f9b1089d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:31:26.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:26.346+0000 7f9b1089d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:31:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:26.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:26.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:26.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:26.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:27.527 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[24773]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[24773]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[29473]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[29473]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:31:27.528 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:27.568 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:27.272+0000 7f9b1089d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:31:27.568 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:27.566+0000 7f9b1089d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:31:27.568 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-mon[25643]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:27.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-mon[25643]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:27.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:31:27.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:31:27.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:27.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:27.659+0000 7f9b1089d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:31:27.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:31:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:31:27.806+0000 7f9b1089d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:31:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: Standby manager daemon x started 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/254084790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]: dispatch 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]: dispatch 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]': finished 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:31:28.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:28 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: Standby manager daemon x started 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/254084790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]': finished 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: Standby manager daemon x started 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4253714442' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/254084790' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]: dispatch 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]: dispatch 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b"}]': finished 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: osdmap e5: 1 total, 0 up, 1 in 2024-03-12T03:31:28.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:28 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:29 smithi153 ceph-mon[25643]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:29 smithi153 ceph-mon[25643]: mgrmap e14: y(active, since 62s), standbys: x 2024-03-12T03:31:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:29 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:31:29.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:29 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3976045982' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:29.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[24773]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[24773]: mgrmap e14: y(active, since 62s), standbys: x 2024-03-12T03:31:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:31:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3976045982' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[29473]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[29473]: mgrmap e14: y(active, since 62s), standbys: x 2024-03-12T03:31:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:31:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:29 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3976045982' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:31.616 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:31 smithi098 ceph-mon[24773]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:31.616 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:31 smithi098 ceph-mon[29473]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:31 smithi153 ceph-mon[25643]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:33.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:33 smithi098 ceph-mon[24773]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:33.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:33 smithi098 ceph-mon[29473]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:33 smithi153 ceph-mon[25643]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:35.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:35 smithi098 ceph-mon[24773]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:35.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:31:35.553 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:35 smithi098 ceph-mon[29473]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:31:35.553 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:35 smithi153 ceph-mon[25643]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T03:31:35.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:36.642 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:36 smithi153 ceph-mon[25643]: Deploying daemon osd.0 on smithi098 2024-03-12T03:31:36.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:36 smithi098 ceph-mon[24773]: Deploying daemon osd.0 on smithi098 2024-03-12T03:31:36.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:36 smithi098 ceph-mon[29473]: Deploying daemon osd.0 on smithi098 2024-03-12T03:31:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:37 smithi153 ceph-mon[25643]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:37.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:37 smithi098 ceph-mon[24773]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:37.705 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:37 smithi098 ceph-mon[29473]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:39.410 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:39 smithi098 ceph-mon[24773]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:39.411 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:39 smithi098 ceph-mon[29473]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:39 smithi153 ceph-mon[25643]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:40 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:40 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:40 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:40 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:40.776 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:40 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:41.684 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:41 smithi098 ceph-mon[24773]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:41.685 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:41 smithi098 ceph-mon[29473]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:41 smithi153 ceph-mon[25643]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:42.658 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 0 on host 'smithi098' 2024-03-12T03:31:42.950 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:42.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:31:42.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:31:42.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:42.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:42.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:42.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:42 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:42 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:31:43.199 DEBUG:teuthology.orchestra.run.smithi098:osd.0> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.0.service 2024-03-12T03:31:43.242 INFO:tasks.cephadm:Deploying osd.1 on smithi098 with /dev/vg_nvme/lv_3... 2024-03-12T03:31:43.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:31:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[29473]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:31:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[29473]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:31:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[29473]: from='osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:31:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[29473]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:31:44.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[24773]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:31:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[24773]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:31:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[24773]: from='osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:31:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-mon[24773]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:31:44.575 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[32344]: 2024-03-12T03:31:44.318+0000 7f68a2cc6700 -1 osd.0 0 waiting for initial osdmap 2024-03-12T03:31:44.575 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:31:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[32344]: 2024-03-12T03:31:44.320+0000 7f689ce5c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:31:44.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:44 smithi153 ceph-mon[25643]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T03:31:44.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:44 smithi153 ceph-mon[25643]: osdmap e6: 1 total, 0 up, 1 in 2024-03-12T03:31:44.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:44 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:44.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:44 smithi153 ceph-mon[25643]: from='osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:31:44.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:44 smithi153 ceph-mon[25643]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:31:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:45 smithi153 ceph-mon[25643]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:31:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:45 smithi153 ceph-mon[25643]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:31:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:45 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:45 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:45 smithi153 ceph-mon[25643]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:45.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[24773]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:31:45.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[24773]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:31:45.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:45.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:45.778 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[24773]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:45.778 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[29473]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:31:45.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[29473]: osdmap e7: 1 total, 0 up, 1 in 2024-03-12T03:31:45.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:45.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:45.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:45 smithi098 ceph-mon[29473]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-12T03:31:46.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:46 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:31:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:46 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:31:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:46 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:46 smithi153 ceph-mon[25643]: osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165] boot 2024-03-12T03:31:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:46 smithi153 ceph-mon[25643]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:31:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:46 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:46.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:31:46.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:31:46.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:46.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[24773]: osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165] boot 2024-03-12T03:31:46.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[24773]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:31:46.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:46.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:31:46.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:31:46.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:46.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[29473]: osd.0 [v2:172.21.15.98:6802/1843576165,v1:172.21.15.98:6803/1843576165] boot 2024-03-12T03:31:46.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[29473]: osdmap e8: 1 total, 1 up, 1 in 2024-03-12T03:31:46.805 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:46 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:31:46.805 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:31:46.805 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:31:46.805 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-12T03:31:46.805 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-12T03:31:46.806 INFO:teuthology.orchestra.run.smithi098.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013844 s, 757 MB/s 2024-03-12T03:31:46.806 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-12T03:31:47.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[24773]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[24773]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[29473]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[29473]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:47 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.570 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_3 2024-03-12T03:31:47.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:47 smithi153 ceph-mon[25643]: osdmap e9: 1 total, 1 up, 1 in 2024-03-12T03:31:47.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:47 smithi153 ceph-mon[25643]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:47.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:47 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:47 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:31:47.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:47 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:47.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:47 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:31:48.496 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:48 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi098 2024-03-12T03:31:48.496 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:48 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi098 to 16064M 2024-03-12T03:31:48.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:48 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi098 2024-03-12T03:31:48.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:48 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi098 to 16064M 2024-03-12T03:31:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:48 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi098 2024-03-12T03:31:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:48 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi098 to 16064M 2024-03-12T03:31:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:49 smithi153 ceph-mon[25643]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:49.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:49 smithi098 ceph-mon[24773]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:49.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:49 smithi098 ceph-mon[29473]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:50.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:50 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:50 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:50 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:50 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:50 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:50 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:50 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:31:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:50 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:31:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:50 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:51 smithi153 ceph-mon[25643]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:51 smithi153 ceph-mon[25643]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:51 smithi098 ceph-mon[24773]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:51 smithi098 ceph-mon[24773]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:51.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:51 smithi098 ceph-mon[29473]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:31:51.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:51 smithi098 ceph-mon[29473]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:52 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3424229954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]: dispatch 2024-03-12T03:31:52.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:52 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]: dispatch 2024-03-12T03:31:52.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:52 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]': finished 2024-03-12T03:31:52.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:52 smithi153 ceph-mon[25643]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:31:52.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:52 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:31:52.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:52 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3418318366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3424229954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]: dispatch 2024-03-12T03:31:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]: dispatch 2024-03-12T03:31:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]': finished 2024-03-12T03:31:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[24773]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:31:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:31:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3418318366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:52.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3424229954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]: dispatch 2024-03-12T03:31:52.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]: dispatch 2024-03-12T03:31:52.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "01a82066-05c4-4819-b27f-4aa3c611eaff"}]': finished 2024-03-12T03:31:52.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[29473]: osdmap e10: 2 total, 1 up, 2 in 2024-03-12T03:31:52.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:31:52.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:52 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3418318366' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:31:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:53 smithi153 ceph-mon[25643]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:53.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:53 smithi098 ceph-mon[24773]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:53 smithi098 ceph-mon[29473]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:55.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:55 smithi098 ceph-mon[24773]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:55 smithi098 ceph-mon[29473]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:55 smithi153 ceph-mon[25643]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:57 smithi098 ceph-mon[24773]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:57 smithi098 ceph-mon[29473]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:57 smithi153 ceph-mon[25643]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:31:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:58.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:31:58.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:58.731 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T03:31:58.731 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:31:59.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:59 smithi098 ceph-mon[24773]: Deploying daemon osd.1 on smithi098 2024-03-12T03:31:59.557 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:31:59 smithi098 ceph-mon[24773]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:59.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:59 smithi098 ceph-mon[29473]: Deploying daemon osd.1 on smithi098 2024-03-12T03:31:59.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:31:59 smithi098 ceph-mon[29473]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:31:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:59 smithi153 ceph-mon[25643]: Deploying daemon osd.1 on smithi098 2024-03-12T03:31:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:31:59 smithi153 ceph-mon[25643]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:01 smithi098 ceph-mon[24773]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:01 smithi098 ceph-mon[29473]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:01 smithi153 ceph-mon[25643]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[24773]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[29473]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:03.523 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:03.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:03 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:03 smithi153 ceph-mon[25643]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:03 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:03 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:03.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:03 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:03.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:03 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:05.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:05 smithi153 ceph-mon[25643]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:05.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:05.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:05.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:05 smithi098 ceph-mon[24773]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:05.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:05.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:05 smithi098 ceph-mon[29473]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:06.139 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 1 on host 'smithi098' 2024-03-12T03:32:06.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:06 smithi153 ceph-mon[25643]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:32:06.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:06.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:06.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:06.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:06.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[24773]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:32:06.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:06.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:06.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:06.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:06.715 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[29473]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T03:32:06.716 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:06.716 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:06.716 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:06.716 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:06.737 DEBUG:teuthology.orchestra.run.smithi098:osd.1> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.1.service 2024-03-12T03:32:06.739 INFO:tasks.cephadm:Deploying osd.2 on smithi098 with /dev/vg_nvme/lv_2... 2024-03-12T03:32:06.739 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:32:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[24773]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[24773]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:32:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[24773]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:32:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[24773]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:07.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[29473]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:07.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[29473]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:32:07.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[29473]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:32:07.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:07.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-mon[29473]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:07.575 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[35306]: 2024-03-12T03:32:07.405+0000 7f34d99d5700 -1 osd.1 0 waiting for initial osdmap 2024-03-12T03:32:07.575 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:07 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[35306]: 2024-03-12T03:32:07.410+0000 7f34d2368700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:32:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:07 smithi153 ceph-mon[25643]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:07 smithi153 ceph-mon[25643]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T03:32:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:07 smithi153 ceph-mon[25643]: osdmap e11: 2 total, 1 up, 2 in 2024-03-12T03:32:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:07 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:07 smithi153 ceph-mon[25643]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:08.626 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[29473]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:08.626 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[29473]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:32:08.626 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:08.626 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:08.626 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[24773]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:08.627 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[24773]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:32:08.627 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:08.627 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:08 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:08 smithi153 ceph-mon[25643]: from='osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:08 smithi153 ceph-mon[25643]: osdmap e12: 2 total, 1 up, 2 in 2024-03-12T03:32:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:08 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:08 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:09.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:32:09.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:32:09.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[24773]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:09.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:09.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[24773]: osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253] boot 2024-03-12T03:32:09.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[24773]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:32:09.490 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:09.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:32:09.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:32:09.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[29473]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:09.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:09.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[29473]: osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253] boot 2024-03-12T03:32:09.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[29473]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:32:09.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:09 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:09 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:32:09.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:09 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:32:09.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:09 smithi153 ceph-mon[25643]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-12T03:32:09.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:09 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:09.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:09 smithi153 ceph-mon[25643]: osd.1 [v2:172.21.15.98:6810/1117288253,v1:172.21.15.98:6811/1117288253] boot 2024-03-12T03:32:09.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:09 smithi153 ceph-mon[25643]: osdmap e13: 2 total, 2 up, 2 in 2024-03-12T03:32:09.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:09 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:32:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:32:10.112 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:32:10.112 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-12T03:32:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-12T03:32:10.113 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123859 s, 847 MB/s 2024-03-12T03:32:10.113 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-12T03:32:10.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:10 smithi153 ceph-mon[25643]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:32:10.789 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:10 smithi098 ceph-mon[24773]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:32:10.789 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:10 smithi098 ceph-mon[29473]: osdmap e14: 2 total, 2 up, 2 in 2024-03-12T03:32:10.812 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_2 2024-03-12T03:32:11.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[24773]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:11.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:11.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:11.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.459 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.459 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[29473]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:11.460 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.460 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:11.460 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:11.460 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.460 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:11 smithi153 ceph-mon[25643]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:12.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:12 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi098 2024-03-12T03:32:12.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:12 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi098 to 8032M 2024-03-12T03:32:12.517 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:12 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi098 2024-03-12T03:32:12.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:12 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi098 to 8032M 2024-03-12T03:32:12.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:12 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi098 2024-03-12T03:32:12.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:12 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi098 to 8032M 2024-03-12T03:32:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:13 smithi153 ceph-mon[25643]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:13.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:13.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:13 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:13.740 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[24773]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:13.740 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:13.740 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:13.740 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:13.740 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[29473]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:13.740 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:13.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:13.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:13 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:14.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:14 smithi098 ceph-mon[24773]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:14.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:14 smithi098 ceph-mon[29473]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:14 smithi153 ceph-mon[25643]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:15 smithi153 ceph-mon[25643]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:15 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3515650124' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70b5963e-5214-4b47-a00f-d25a8237c8a5"}]: dispatch 2024-03-12T03:32:15.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:15 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3515650124' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70b5963e-5214-4b47-a00f-d25a8237c8a5"}]': finished 2024-03-12T03:32:15.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:15 smithi153 ceph-mon[25643]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:32:15.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:15 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:15.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[24773]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3515650124' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70b5963e-5214-4b47-a00f-d25a8237c8a5"}]: dispatch 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3515650124' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70b5963e-5214-4b47-a00f-d25a8237c8a5"}]': finished 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[24773]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[29473]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3515650124' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "70b5963e-5214-4b47-a00f-d25a8237c8a5"}]: dispatch 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3515650124' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "70b5963e-5214-4b47-a00f-d25a8237c8a5"}]': finished 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[29473]: osdmap e15: 3 total, 2 up, 3 in 2024-03-12T03:32:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:15 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:16.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:16 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1113199299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:16 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1113199299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:16.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:16 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1113199299' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:17.804 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:17 smithi098 ceph-mon[24773]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:17.804 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:17 smithi098 ceph-mon[29473]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:17 smithi153 ceph-mon[25643]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:19 smithi153 ceph-mon[25643]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:19.772 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:19 smithi098 ceph-mon[24773]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:19.773 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:19 smithi098 ceph-mon[29473]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:21.434 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:21 smithi098 ceph-mon[24773]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:21.435 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:21 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:32:21.435 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:21 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:21.435 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:21 smithi098 ceph-mon[29473]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:21.435 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:21 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:32:21.435 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:21 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:21 smithi153 ceph-mon[25643]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:21 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T03:32:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:21 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:22.633 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:22 smithi098 ceph-mon[24773]: Deploying daemon osd.2 on smithi098 2024-03-12T03:32:22.634 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:22 smithi098 ceph-mon[29473]: Deploying daemon osd.2 on smithi098 2024-03-12T03:32:22.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:22 smithi153 ceph-mon[25643]: Deploying daemon osd.2 on smithi098 2024-03-12T03:32:23.634 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:23 smithi098 ceph-mon[24773]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:23.634 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:23 smithi098 ceph-mon[29473]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:23 smithi153 ceph-mon[25643]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:25.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:25 smithi098 ceph-mon[24773]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:25.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:25 smithi098 ceph-mon[29473]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:25 smithi153 ceph-mon[25643]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:27 smithi153 ceph-mon[25643]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:32:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:32:27.551 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:27.551 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:27.551 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:27.551 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:27.551 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[24773]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:27.551 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:32:27.551 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:32:27.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:27.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:27.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:27.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:27.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[29473]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:27.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:32:27.552 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:32:29.442 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 2 on host 'smithi098' 2024-03-12T03:32:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:29 smithi153 ceph-mon[25643]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:29 smithi153 ceph-mon[25643]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:32:29.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:29 smithi098 ceph-mon[24773]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:29.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:29 smithi098 ceph-mon[24773]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:32:29.715 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:29 smithi098 ceph-mon[29473]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:29.715 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:29 smithi098 ceph-mon[29473]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T03:32:30.004 DEBUG:teuthology.orchestra.run.smithi098:osd.2> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.2.service 2024-03-12T03:32:30.007 INFO:tasks.cephadm:Deploying osd.3 on smithi098 with /dev/vg_nvme/lv_1... 2024-03-12T03:32:30.007 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T03:32:30.560 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[29473]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:32:30.560 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[29473]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:32:30.560 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:30.560 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[29473]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:30.560 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:30.560 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:30.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[24773]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:32:30.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[24773]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:32:30.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:30.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[24773]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:30.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:30.561 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:30.561 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[38254]: 2024-03-12T03:32:30.419+0000 7feeebf90700 -1 osd.2 0 waiting for initial osdmap 2024-03-12T03:32:30.561 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[38254]: 2024-03-12T03:32:30.426+0000 7feee7128700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:32:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:30 smithi153 ceph-mon[25643]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T03:32:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:30 smithi153 ceph-mon[25643]: osdmap e16: 3 total, 2 up, 3 in 2024-03-12T03:32:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:30 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:30 smithi153 ceph-mon[25643]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:30.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:30 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:30.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:30 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:31.564 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[24773]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:31.564 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[24773]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:31.564 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[24773]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:32:31.564 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:31.564 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:31.564 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[29473]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:31.564 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[29473]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:31.565 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[29473]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:32:31.565 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:31.565 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:31 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:31 smithi153 ceph-mon[25643]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:31 smithi153 ceph-mon[25643]: from='osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:31 smithi153 ceph-mon[25643]: osdmap e17: 3 total, 2 up, 3 in 2024-03-12T03:32:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:31 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:31 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:32.776 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T03:32:32.776 INFO:teuthology.orchestra.run.smithi098.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T03:32:32.776 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10+0 records in 2024-03-12T03:32:32.776 INFO:teuthology.orchestra.run.smithi098.stderr:10+0 records out 2024-03-12T03:32:32.776 INFO:teuthology.orchestra.run.smithi098.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132361 s, 792 MB/s 2024-03-12T03:32:32.776 INFO:teuthology.orchestra.run.smithi098.stderr:--> Zapping successful for: 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[24773]: osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193] boot 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[24773]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:32:32.875 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:32.876 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[29473]: osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193] boot 2024-03-12T03:32:32.876 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[29473]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:32:32.876 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:32 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:32.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:32 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:32:32.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:32 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:32:32.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:32 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:32.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:32 smithi153 ceph-mon[25643]: osd.2 [v2:172.21.15.98:6818/3197862193,v1:172.21.15.98:6819/3197862193] boot 2024-03-12T03:32:32.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:32 smithi153 ceph-mon[25643]: osdmap e18: 3 total, 3 up, 3 in 2024-03-12T03:32:32.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:32 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:32:33.549 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi098:vg_nvme/lv_1 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[24773]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[24773]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[29473]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-12T03:32:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[29473]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:32:33.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:33 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:32:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:33 smithi153 ceph-mon[25643]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-12T03:32:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:33 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-12T03:32:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:33 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-12T03:32:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:33 smithi153 ceph-mon[25643]: osdmap e19: 3 total, 3 up, 3 in 2024-03-12T03:32:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:33 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-12T03:32:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:34 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:32:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:34 smithi098 ceph-mon[24773]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:32:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:34 smithi098 ceph-mon[24773]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:34.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:34 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:32:34.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:34 smithi098 ceph-mon[29473]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:32:34.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:34 smithi098 ceph-mon[29473]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:34.825 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:32:34 smithi098 sudo[39530]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:32:34.825 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:32:34 smithi098 sudo[39530]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:34.825 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:32:34 smithi098 sudo[39530]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:34 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-12T03:32:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:34 smithi153 ceph-mon[25643]: osdmap e20: 3 total, 3 up, 3 in 2024-03-12T03:32:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:34 smithi153 ceph-mon[25643]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:35.165 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:32:34 smithi098 sudo[39545]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:32:35.165 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:32:34 smithi098 sudo[39545]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:35.165 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 03:32:34 smithi098 sudo[39545]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:35.425 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39551]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:32:35.425 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39551]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:35.425 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39551]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:35.684 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39557]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:32:35.685 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39557]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:35.685 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39557]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:35.685 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39560]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:32:35.685 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39560]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:35.685 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39560]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:35.685 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi098 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi098 to 5354M 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:35.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi098 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi098 to 5354M 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:35.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:35.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:35.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: osdmap e21: 3 total, 3 up, 3 in 2024-03-12T03:32:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi098 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi098 to 5354M 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:36.024 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39582]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:32:36.024 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39582]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:36.024 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 03:32:35 smithi098 sudo[39582]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:36.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39591]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:32:36.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39591]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:36.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39591]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39615]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:32:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39615]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:36.704 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39615]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:36.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39675]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:32:36.955 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39675]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:36.955 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39675]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:37.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39704]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:32:37.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39704]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:37.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:36 smithi098 sudo[39704]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:37.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 sudo[27164]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-12T03:32:37.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 sudo[27164]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:37.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 sudo[27164]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:37.731 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:37.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:37.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:37.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:37.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:37.733 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 sudo[27167]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 sudo[27167]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 sudo[27167]: pam_unix(sudo:session): session closed for user root 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: pgmap v76: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi098:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:37.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:38.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:38.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:38.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:38.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:38.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:38.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[24773]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:38.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:38 smithi098 ceph-mon[29473]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:38.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:38.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-12T03:32:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:32:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:32:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:32:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-12T03:32:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:38 smithi153 ceph-mon[25643]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3175714854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c87f9327-c5f5-49b3-9505-2287534ab2b6"}]: dispatch 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3175714854' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c87f9327-c5f5-49b3-9505-2287534ab2b6"}]': finished 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[24773]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/438273160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3175714854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c87f9327-c5f5-49b3-9505-2287534ab2b6"}]: dispatch 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3175714854' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c87f9327-c5f5-49b3-9505-2287534ab2b6"}]': finished 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[29473]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T03:32:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:39.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:39 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/438273160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:39 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3175714854' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c87f9327-c5f5-49b3-9505-2287534ab2b6"}]: dispatch 2024-03-12T03:32:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:39 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3175714854' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c87f9327-c5f5-49b3-9505-2287534ab2b6"}]': finished 2024-03-12T03:32:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:39 smithi153 ceph-mon[25643]: osdmap e22: 4 total, 3 up, 4 in 2024-03-12T03:32:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:39 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:39.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:39 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/438273160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:32:40.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:40 smithi098 ceph-mon[24773]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.8 KiB/s rd, 230 KiB/s wr, 15 op/s 2024-03-12T03:32:40.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:40 smithi098 ceph-mon[29473]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.8 KiB/s rd, 230 KiB/s wr, 15 op/s 2024-03-12T03:32:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:40 smithi153 ceph-mon[25643]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 4.8 KiB/s rd, 230 KiB/s wr, 15 op/s 2024-03-12T03:32:43.665 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:43 smithi098 ceph-mon[24773]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-12T03:32:43.666 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:43 smithi098 ceph-mon[29473]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-12T03:32:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:43 smithi153 ceph-mon[25643]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 170 KiB/s wr, 11 op/s 2024-03-12T03:32:45.617 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:45 smithi098 ceph-mon[29473]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-03-12T03:32:45.617 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:45 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:32:45.617 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:45 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:45.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:45 smithi098 ceph-mon[24773]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-03-12T03:32:45.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:32:45.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:45 smithi153 ceph-mon[25643]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 135 KiB/s wr, 9 op/s 2024-03-12T03:32:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:45 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T03:32:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:45 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:46.545 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:46 smithi098 ceph-mon[24773]: Deploying daemon osd.3 on smithi098 2024-03-12T03:32:46.546 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:46 smithi098 ceph-mon[29473]: Deploying daemon osd.3 on smithi098 2024-03-12T03:32:46.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:46 smithi153 ceph-mon[25643]: Deploying daemon osd.3 on smithi098 2024-03-12T03:32:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:47 smithi153 ceph-mon[25643]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-12T03:32:47.698 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:47 smithi098 ceph-mon[24773]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-12T03:32:47.699 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:47 smithi098 ceph-mon[29473]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-12T03:32:49.649 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:49 smithi098 ceph-mon[24773]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-12T03:32:49.649 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:49 smithi098 ceph-mon[29473]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-12T03:32:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:49 smithi153 ceph-mon[25643]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-03-12T03:32:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:50 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:50 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:50 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:50 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:50 smithi153 ceph-mon[25643]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 113 KiB/s wr, 7 op/s 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[24773]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 113 KiB/s wr, 7 op/s 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:50 smithi098 ceph-mon[29473]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 113 KiB/s wr, 7 op/s 2024-03-12T03:32:53.286 INFO:teuthology.orchestra.run.smithi098.stdout:Created osd(s) 3 on host 'smithi098' 2024-03-12T03:32:53.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[24773]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[24773]: from='osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:32:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[24773]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:32:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[29473]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[29473]: from='osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:32:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[29473]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:32:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:53 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:53 smithi153 ceph-mon[25643]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:53 smithi153 ceph-mon[25643]: from='osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:32:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:53 smithi153 ceph-mon[25643]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T03:32:53.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:53 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:53.946 DEBUG:teuthology.orchestra.run.smithi098:osd.3> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.3.service 2024-03-12T03:32:53.948 INFO:tasks.cephadm:Deploying osd.4 on smithi153 with /dev/vg_nvme/lv_4... 2024-03-12T03:32:53.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-12T03:32:54.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:54 smithi153 ceph-mon[25643]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:32:54.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:54 smithi153 ceph-mon[25643]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:32:54.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:54 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:54.518 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:54 smithi153 ceph-mon[25643]: from='osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:54.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:54 smithi153 ceph-mon[25643]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:54.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:54 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[24773]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[24773]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[24773]: from='osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[24773]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[29473]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T03:32:54.793 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[29473]: osdmap e23: 4 total, 3 up, 4 in 2024-03-12T03:32:54.794 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:54.794 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[29473]: from='osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:54.794 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[29473]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T03:32:54.794 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:54.794 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[41222]: 2024-03-12T03:32:54.423+0000 7fb0289fb700 -1 osd.3 0 waiting for initial osdmap 2024-03-12T03:32:54.794 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 03:32:54 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[41222]: 2024-03-12T03:32:54.426+0000 7fb023b93700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:32:55.552 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:55 smithi153 ceph-mon[25643]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:55.552 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:55 smithi153 ceph-mon[25643]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:55.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:55 smithi153 ceph-mon[25643]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:32:55.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:55 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:55.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:55 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:55.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[24773]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:55.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[24773]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[24773]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[29473]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[29473]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]': finished 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[29473]: osdmap e24: 4 total, 3 up, 4 in 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:55.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:55 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:55.917 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-12T03:32:55.917 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-12T03:32:55.917 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-12T03:32:55.917 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-12T03:32:55.917 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115139 s, 911 MB/s 2024-03-12T03:32:55.917 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-12T03:32:56.479 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-03-12T03:32:56.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:56 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:32:56.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:56 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:32:56.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:56 smithi153 ceph-mon[25643]: osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879] boot 2024-03-12T03:32:56.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:56 smithi153 ceph-mon[25643]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T03:32:56.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:56 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:56.796 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:32:56.796 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:32:56.796 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[24773]: osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879] boot 2024-03-12T03:32:56.796 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[24773]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T03:32:56.796 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:56.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:32:56.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:32:56.797 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[29473]: osd.3 [v2:172.21.15.98:6826/3388374879,v1:172.21.15.98:6827/3388374879] boot 2024-03-12T03:32:56.797 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[29473]: osdmap e25: 4 total, 4 up, 4 in 2024-03-12T03:32:56.797 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:56 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:32:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:57 smithi098 ceph-mon[24773]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:57 smithi098 ceph-mon[24773]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T03:32:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:57 smithi098 ceph-mon[29473]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:57 smithi098 ceph-mon[29473]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T03:32:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:57 smithi153 ceph-mon[25643]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-03-12T03:32:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:57 smithi153 ceph-mon[25643]: osdmap e26: 4 total, 4 up, 4 in 2024-03-12T03:32:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi098 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi098 to 4016M 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:58.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:58.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:58.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:58.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:58.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:32:58 smithi153 ceph-mon[25643]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi098 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi098 to 4016M 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[29473]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi098 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi098 to 4016M 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='client.34187 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:32:58.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:32:58 smithi098 ceph-mon[24773]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:01 smithi153 ceph-mon[25643]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:01 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/2280208311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]: dispatch 2024-03-12T03:33:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:01 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]: dispatch 2024-03-12T03:33:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:01 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]': finished 2024-03-12T03:33:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:01 smithi153 ceph-mon[25643]: osdmap e27: 5 total, 4 up, 5 in 2024-03-12T03:33:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:01 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:01 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/3591776253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[24773]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/2280208311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]: dispatch 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]: dispatch 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]': finished 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[24773]: osdmap e27: 5 total, 4 up, 5 in 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/3591776253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:01.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[29473]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:01.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/2280208311' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]: dispatch 2024-03-12T03:33:01.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]: dispatch 2024-03-12T03:33:01.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73791fe8-e473-4d42-ad03-c32c85bc5738"}]': finished 2024-03-12T03:33:01.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[29473]: osdmap e27: 5 total, 4 up, 5 in 2024-03-12T03:33:01.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:01.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:01 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/3591776253' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:03.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:03 smithi153 ceph-mon[25643]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:03.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:03 smithi098 ceph-mon[24773]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:03 smithi098 ceph-mon[29473]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:05.669 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:05 smithi153 ceph-mon[25643]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:05.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:05 smithi098 ceph-mon[24773]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:05 smithi098 ceph-mon[29473]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:06.634 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:33:06.634 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:33:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T03:33:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:07.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:07 smithi153 ceph-mon[25643]: Deploying daemon osd.4 on smithi153 2024-03-12T03:33:07.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:07 smithi153 ceph-mon[25643]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:07 smithi098 ceph-mon[24773]: Deploying daemon osd.4 on smithi153 2024-03-12T03:33:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:07 smithi098 ceph-mon[24773]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:07 smithi098 ceph-mon[29473]: Deploying daemon osd.4 on smithi153 2024-03-12T03:33:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:07 smithi098 ceph-mon[29473]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:09.523 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:09 smithi153 ceph-mon[25643]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:09.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:09 smithi098 ceph-mon[24773]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:09 smithi098 ceph-mon[29473]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:11.552 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:11 smithi153 ceph-mon[25643]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:11.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:11.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:11.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:11.553 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:11 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[24773]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[29473]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:11 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:13.609 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:13 smithi153 ceph-mon[25643]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:13.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:13 smithi098 ceph-mon[24773]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:13.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:13 smithi098 ceph-mon[29473]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:14.207 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 4 on host 'smithi153' 2024-03-12T03:33:14.794 DEBUG:teuthology.orchestra.run.smithi153:osd.4> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.4.service 2024-03-12T03:33:14.797 INFO:tasks.cephadm:Deploying osd.5 on smithi153 with /dev/vg_nvme/lv_3... 2024-03-12T03:33:14.797 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-12T03:33:15.434 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:15 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:15.434 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:15 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:15.434 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:15 smithi153 ceph-mon[25643]: from='osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:33:15.434 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:15 smithi153 ceph-mon[25643]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:33:15.434 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:15 smithi153 ceph-mon[25643]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:15.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:15.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:15.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[24773]: from='osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:33:15.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[24773]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:33:15.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[24773]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:15.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:15.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:15.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[29473]: from='osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:33:15.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[29473]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T03:33:15.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:15 smithi098 ceph-mon[29473]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:16 smithi153 ceph-mon[25643]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:33:16.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:16 smithi153 ceph-mon[25643]: osdmap e28: 5 total, 4 up, 5 in 2024-03-12T03:33:16.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:16 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:16.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:16 smithi153 ceph-mon[25643]: from='osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:16.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:16 smithi153 ceph-mon[25643]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:16.443 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 03:33:16 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[29021]: 2024-03-12T03:33:16.250+0000 7ff4f2784700 -1 osd.4 0 waiting for initial osdmap 2024-03-12T03:33:16.443 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 03:33:16 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[29021]: 2024-03-12T03:33:16.256+0000 7ff4ef11f700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[24773]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[24773]: osdmap e28: 5 total, 4 up, 5 in 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[24773]: from='osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[24773]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[29473]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[29473]: osdmap e28: 5 total, 4 up, 5 in 2024-03-12T03:33:16.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:16.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[29473]: from='osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:16.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:16 smithi098 ceph-mon[29473]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:16.937 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-12T03:33:16.937 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-12T03:33:16.937 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-12T03:33:16.937 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-12T03:33:16.937 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136208 s, 770 MB/s 2024-03-12T03:33:16.937 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-12T03:33:17.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:17 smithi153 ceph-mon[25643]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:33:17.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:17 smithi153 ceph-mon[25643]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:33:17.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:17 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:17.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:17 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:17.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:17 smithi153 ceph-mon[25643]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:17.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:17 smithi153 ceph-mon[25643]: osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885] boot 2024-03-12T03:33:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[24773]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:33:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[24773]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:33:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[24773]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[24773]: osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885] boot 2024-03-12T03:33:17.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[29473]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:33:17.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[29473]: osdmap e29: 5 total, 4 up, 5 in 2024-03-12T03:33:17.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:17.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:17.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[29473]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-12T03:33:17.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:17 smithi098 ceph-mon[29473]: osd.4 [v2:172.21.15.153:6800/1148111885,v1:172.21.15.153:6801/1148111885] boot 2024-03-12T03:33:17.671 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-03-12T03:33:18.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:18 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:33:18.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:18 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:33:18.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:18 smithi153 ceph-mon[25643]: osdmap e30: 5 total, 5 up, 5 in 2024-03-12T03:33:18.519 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:18 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[24773]: osdmap e30: 5 total, 5 up, 5 in 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[29473]: osdmap e30: 5 total, 5 up, 5 in 2024-03-12T03:33:18.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:18 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:33:19.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:19 smithi153 ceph-mon[25643]: osdmap e31: 5 total, 5 up, 5 in 2024-03-12T03:33:19.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:19 smithi153 ceph-mon[25643]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:19.474 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:19 smithi153 ceph-mon[25643]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:33:19.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:19 smithi098 ceph-mon[24773]: osdmap e31: 5 total, 5 up, 5 in 2024-03-12T03:33:19.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:19 smithi098 ceph-mon[24773]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:19.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:19 smithi098 ceph-mon[24773]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:33:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:19 smithi098 ceph-mon[29473]: osdmap e31: 5 total, 5 up, 5 in 2024-03-12T03:33:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:19 smithi098 ceph-mon[29473]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:19 smithi098 ceph-mon[29473]: osdmap e32: 5 total, 5 up, 5 in 2024-03-12T03:33:20.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi153 2024-03-12T03:33:20.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:20.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi153 to 17088M 2024-03-12T03:33:20.810 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:33:20.811 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:20 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi153 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi153 to 17088M 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi153 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi153 to 17088M 2024-03-12T03:33:20.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: osdmap e33: 5 total, 5 up, 5 in 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: pgmap v110: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='client.34214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:33:20.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:20 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:22.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:22 smithi153 ceph-mon[25643]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:33:22.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:22 smithi098 ceph-mon[24773]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:33:22.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:22 smithi098 ceph-mon[29473]: osdmap e34: 5 total, 5 up, 5 in 2024-03-12T03:33:23.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[24773]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-12T03:33:23.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/2430823940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]: dispatch 2024-03-12T03:33:23.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]: dispatch 2024-03-12T03:33:23.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]': finished 2024-03-12T03:33:23.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[24773]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T03:33:23.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/3600518132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[29473]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/2430823940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]: dispatch 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]: dispatch 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]': finished 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[29473]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:23.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:23 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/3600518132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:23 smithi153 ceph-mon[25643]: pgmap v112: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-12T03:33:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:23 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/2430823940' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]: dispatch 2024-03-12T03:33:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:23 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]: dispatch 2024-03-12T03:33:23.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:23 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d75bfb5-af19-49cb-9c4e-9eae9ddc016e"}]': finished 2024-03-12T03:33:23.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:23 smithi153 ceph-mon[25643]: osdmap e35: 6 total, 5 up, 6 in 2024-03-12T03:33:23.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:23 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:23.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:23 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/3600518132' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:25 smithi153 ceph-mon[25643]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-12T03:33:25.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:25 smithi098 ceph-mon[24773]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-12T03:33:25.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:25 smithi098 ceph-mon[29473]: pgmap v114: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-12T03:33:26.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:33:26 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:33:26.356+0000 7fb8c65b3700 -1 calc_pg_upmaps abort due to max <= 0 2024-03-12T03:33:27.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:27 smithi153 ceph-mon[25643]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:27.686 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:33:27.686 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:33:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:27 smithi098 ceph-mon[24773]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:33:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:33:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:27 smithi098 ceph-mon[29473]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:33:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:33:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:29 smithi153 ceph-mon[25643]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:29 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:33:29.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:29 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:29 smithi098 ceph-mon[24773]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:29 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:33:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:29 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:29 smithi098 ceph-mon[29473]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:29 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T03:33:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:29 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:30 smithi153 ceph-mon[25643]: Deploying daemon osd.5 on smithi153 2024-03-12T03:33:30.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:30 smithi098 ceph-mon[24773]: Deploying daemon osd.5 on smithi153 2024-03-12T03:33:30.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:30 smithi098 ceph-mon[29473]: Deploying daemon osd.5 on smithi153 2024-03-12T03:33:31.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:31 smithi153 ceph-mon[25643]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:31.725 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:31 smithi098 ceph-mon[24773]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:31.725 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:31 smithi098 ceph-mon[29473]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:33 smithi153 ceph-mon[25643]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:33 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:33 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:33 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:33.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:33 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[24773]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[29473]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:33 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:35.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:35 smithi153 ceph-mon[25643]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:35.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:35 smithi098 ceph-mon[24773]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:35.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:35 smithi098 ceph-mon[29473]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:36.624 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 5 on host 'smithi153' 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:36.897 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:36.898 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:36 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:37.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:36 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:37.170 DEBUG:teuthology.orchestra.run.smithi153:osd.5> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.5.service 2024-03-12T03:33:37.172 INFO:tasks.cephadm:Deploying osd.6 on smithi153 with /dev/vg_nvme/lv_2... 2024-03-12T03:33:37.173 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-12T03:33:38.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:37 smithi153 ceph-mon[25643]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:33:38.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:37 smithi153 ceph-mon[25643]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T03:33:38.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:37 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:38.057 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:37 smithi153 ceph-mon[25643]: from='osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:38.058 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:37 smithi153 ceph-mon[25643]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:38.058 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 03:33:37 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[31977]: 2024-03-12T03:33:37.808+0000 7fa73d919700 -1 osd.5 0 waiting for initial osdmap 2024-03-12T03:33:38.058 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 03:33:37 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[31977]: 2024-03-12T03:33:37.813+0000 7fa736aad700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[24773]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[24773]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[24773]: from='osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[24773]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[29473]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[29473]: osdmap e36: 6 total, 5 up, 6 in 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[29473]: from='osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:37 smithi098 ceph-mon[29473]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641] boot 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:33:39.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:33:39.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:39.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641] boot 2024-03-12T03:33:39.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T03:33:39.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:38 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:33:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: osdmap e37: 6 total, 5 up, 6 in 2024-03-12T03:33:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-12T03:33:39.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: osd.5 [v2:172.21.15.153:6808/1227869641,v1:172.21.15.153:6809/1227869641] boot 2024-03-12T03:33:39.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: osdmap e38: 6 total, 6 up, 6 in 2024-03-12T03:33:39.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:38 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:33:39.978 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-12T03:33:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-12T03:33:39.979 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-12T03:33:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-12T03:33:39.979 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140153 s, 748 MB/s 2024-03-12T03:33:39.979 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-12T03:33:40.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:40 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:33:40.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:40 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:33:40.571 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:40 smithi153 ceph-mon[25643]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T03:33:40.596 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-03-12T03:33:40.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:40 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:33:40.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:40 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:33:40.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:40 smithi098 ceph-mon[24773]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T03:33:40.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:40 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:33:40.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:40 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:33:40.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:40 smithi098 ceph-mon[29473]: osdmap e39: 6 total, 6 up, 6 in 2024-03-12T03:33:41.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:41 smithi153 ceph-mon[25643]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:41.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:41 smithi153 ceph-mon[25643]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:33:41.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:41 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:41 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:41.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:41 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:41.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:41 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.569 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:41 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[24773]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[24773]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:33:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[29473]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[29473]: osdmap e40: 6 total, 6 up, 6 in 2024-03-12T03:33:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:33:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:41 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:42.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:42 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi153 2024-03-12T03:33:42.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:42 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi153 to 8544M 2024-03-12T03:33:42.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:42 smithi153 ceph-mon[25643]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:33:42.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:42 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi153 2024-03-12T03:33:42.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:42 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi153 to 8544M 2024-03-12T03:33:42.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:42 smithi098 ceph-mon[24773]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:33:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:42 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi153 2024-03-12T03:33:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:42 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi153 to 8544M 2024-03-12T03:33:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:42 smithi098 ceph-mon[29473]: osdmap e41: 6 total, 6 up, 6 in 2024-03-12T03:33:43.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:43 smithi153 ceph-mon[25643]: pgmap v129: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:43.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:43 smithi153 ceph-mon[25643]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:33:43.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:43 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:33:43.638 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:43 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:33:43.639 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:43 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[29473]: pgmap v129: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[29473]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[24773]: pgmap v129: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[24773]: osdmap e42: 6 total, 6 up, 6 in 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:33:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:43 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:44.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:44 smithi153 ceph-mon[25643]: from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:44.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:44 smithi098 ceph-mon[24773]: from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:44 smithi098 ceph-mon[29473]: from='client.34241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:33:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:45 smithi153 ceph-mon[25643]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:45 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/2634286433' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]: dispatch 2024-03-12T03:33:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:45 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]: dispatch 2024-03-12T03:33:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:45 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]': finished 2024-03-12T03:33:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:45 smithi153 ceph-mon[25643]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T03:33:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:45 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[29473]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/2634286433' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]: dispatch 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]: dispatch 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]': finished 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[29473]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[24773]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/2634286433' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]: dispatch 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]: dispatch 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c82a99fe-44eb-4019-ac81-355318f668ef"}]': finished 2024-03-12T03:33:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[24773]: osdmap e43: 7 total, 6 up, 7 in 2024-03-12T03:33:45.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:45 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:33:46.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:46 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/2931108668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:46.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:46 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/2931108668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:46 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/2931108668' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:33:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:47 smithi153 ceph-mon[25643]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:47 smithi098 ceph-mon[24773]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:47 smithi098 ceph-mon[29473]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:49.671 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:49 smithi153 ceph-mon[25643]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:49.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:49 smithi098 ceph-mon[24773]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:49.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:49 smithi098 ceph-mon[29473]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:51.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:51 smithi153 ceph-mon[25643]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:51.645 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:51 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:33:51.645 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:51 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:51 smithi098 ceph-mon[24773]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:51 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:33:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:51 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:51.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:51 smithi098 ceph-mon[29473]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:51.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:51 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T03:33:51.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:51 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:52 smithi153 ceph-mon[25643]: Deploying daemon osd.6 on smithi153 2024-03-12T03:33:52.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:52 smithi098 ceph-mon[29473]: Deploying daemon osd.6 on smithi153 2024-03-12T03:33:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:52 smithi098 ceph-mon[24773]: Deploying daemon osd.6 on smithi153 2024-03-12T03:33:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:53 smithi153 ceph-mon[25643]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:53.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:53 smithi098 ceph-mon[24773]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:53 smithi098 ceph-mon[29473]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:55.557 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:55 smithi153 ceph-mon[25643]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:55 smithi098 ceph-mon[24773]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:55 smithi098 ceph-mon[29473]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:57.399 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:57 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:57.399 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:57 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:57.400 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:57 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:57.400 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:57 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:57.400 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:57 smithi153 ceph-mon[25643]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[24773]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:33:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:33:57.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:33:57.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:57 smithi098 ceph-mon[29473]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:59.153 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 6 on host 'smithi153' 2024-03-12T03:33:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:59 smithi153 ceph-mon[25643]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:59 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:59 smithi153 ceph-mon[25643]: from='osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:33:59.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:33:59 smithi153 ceph-mon[25643]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:33:59.784 DEBUG:teuthology.orchestra.run.smithi153:osd.6> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6.service 2024-03-12T03:33:59.787 INFO:tasks.cephadm:Deploying osd.7 on smithi153 with /dev/vg_nvme/lv_1... 2024-03-12T03:33:59.787 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[24773]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[24773]: from='osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[24773]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[29473]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[29473]: from='osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:33:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:33:59 smithi098 ceph-mon[29473]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T03:34:01.161 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:00 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:01.161 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:00 smithi153 ceph-mon[25643]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:34:01.162 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:00 smithi153 ceph-mon[25643]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T03:34:01.162 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:00 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:01.162 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:00 smithi153 ceph-mon[25643]: from='osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:01.162 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:00 smithi153 ceph-mon[25643]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:01.162 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:00 smithi153 ceph-mon[25643]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:34:01.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[24773]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[24773]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[24773]: from='osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[24773]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[24773]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[29473]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[29473]: osdmap e44: 7 total, 6 up, 7 in 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[29473]: from='osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[29473]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:01.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:00 smithi098 ceph-mon[29473]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:34:01.443 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 03:34:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[34867]: 2024-03-12T03:34:01.158+0000 7f30bab22700 -1 osd.6 0 waiting for initial osdmap 2024-03-12T03:34:01.443 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 03:34:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[34867]: 2024-03-12T03:34:01.166+0000 7f30b74bd700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:34:01.699 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-12T03:34:01.699 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-12T03:34:01.699 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-03-12T03:34:01.699 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-03-12T03:34:01.699 INFO:teuthology.orchestra.run.smithi153.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0138982 s, 754 MB/s 2024-03-12T03:34:01.699 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-03-12T03:34:02.395 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:02 smithi153 ceph-mon[25643]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:34:02.395 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:02 smithi153 ceph-mon[25643]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T03:34:02.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:02 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:02.396 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:02 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[24773]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[24773]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[29473]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[29473]: osdmap e45: 7 total, 6 up, 7 in 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:02 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:02.689 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi153:vg_nvme/lv_1 2024-03-12T03:34:03.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:03 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:34:03.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:03 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:34:03.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:03 smithi153 ceph-mon[25643]: osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115] boot 2024-03-12T03:34:03.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:03 smithi153 ceph-mon[25643]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T03:34:03.414 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:03 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:03.415 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:03 smithi153 ceph-mon[25643]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:34:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[24773]: osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115] boot 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[24773]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[24773]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[29473]: osd.6 [v2:172.21.15.153:6816/287724115,v1:172.21.15.153:6817/287724115] boot 2024-03-12T03:34:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[29473]: osdmap e46: 7 total, 7 up, 7 in 2024-03-12T03:34:03.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:03.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:03 smithi098 ceph-mon[29473]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-12T03:34:04.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:04 smithi153 ceph-mon[25643]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T03:34:04.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:04 smithi098 ceph-mon[24773]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T03:34:04.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:04 smithi098 ceph-mon[29473]: osdmap e47: 7 total, 7 up, 7 in 2024-03-12T03:34:05.438 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: osdmap e48: 7 total, 7 up, 7 in 2024-03-12T03:34:05.438 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:05.438 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:05.439 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:05 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: osdmap e48: 7 total, 7 up, 7 in 2024-03-12T03:34:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: osdmap e48: 7 total, 7 up, 7 in 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:05.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:05.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:05.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:05.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:05.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:05 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:06 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi153 2024-03-12T03:34:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:06 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi153 to 5696M 2024-03-12T03:34:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:06 smithi153 ceph-mon[25643]: osdmap e49: 7 total, 7 up, 7 in 2024-03-12T03:34:06.347 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:34:06.348 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:34:06.348 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:06 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi153 2024-03-12T03:34:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi153 to 5696M 2024-03-12T03:34:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[24773]: osdmap e49: 7 total, 7 up, 7 in 2024-03-12T03:34:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:34:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:34:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:06.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi153 2024-03-12T03:34:06.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi153 to 5696M 2024-03-12T03:34:06.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[29473]: osdmap e49: 7 total, 7 up, 7 in 2024-03-12T03:34:06.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-12T03:34:06.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-12T03:34:06.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:06 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:07.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:07 smithi153 ceph-mon[25643]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:07.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:07 smithi153 ceph-mon[25643]: osdmap e50: 7 total, 7 up, 7 in 2024-03-12T03:34:07.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:07 smithi153 ceph-mon[25643]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:07.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:07 smithi098 ceph-mon[24773]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:07 smithi098 ceph-mon[24773]: osdmap e50: 7 total, 7 up, 7 in 2024-03-12T03:34:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:07 smithi098 ceph-mon[24773]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:07.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:07 smithi098 ceph-mon[29473]: from='client.34268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:07.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:07 smithi098 ceph-mon[29473]: osdmap e50: 7 total, 7 up, 7 in 2024-03-12T03:34:07.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:07 smithi098 ceph-mon[29473]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:08.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/3951741341' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]: dispatch 2024-03-12T03:34:08.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]: dispatch 2024-03-12T03:34:08.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[24773]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]': finished 2024-03-12T03:34:08.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[24773]: osdmap e51: 8 total, 7 up, 8 in 2024-03-12T03:34:08.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:08.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/3951741341' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]: dispatch 2024-03-12T03:34:08.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]: dispatch 2024-03-12T03:34:08.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[29473]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]': finished 2024-03-12T03:34:08.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[29473]: osdmap e51: 8 total, 7 up, 8 in 2024-03-12T03:34:08.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:08 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:08 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/3951741341' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]: dispatch 2024-03-12T03:34:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:08 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]: dispatch 2024-03-12T03:34:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:08 smithi153 ceph-mon[25643]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "abaf358b-9e89-468f-a1d5-3ed04d8ad102"}]': finished 2024-03-12T03:34:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:08 smithi153 ceph-mon[25643]: osdmap e51: 8 total, 7 up, 8 in 2024-03-12T03:34:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:08 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:09 smithi153 ceph-mon[25643]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-12T03:34:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:09 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/884889646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:34:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:09 smithi098 ceph-mon[24773]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-12T03:34:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:09 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/884889646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:34:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:09 smithi098 ceph-mon[29473]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-12T03:34:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:09 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/884889646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-12T03:34:11.524 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:11 smithi153 ceph-mon[25643]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-12T03:34:11.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:11 smithi098 ceph-mon[24773]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-12T03:34:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:11 smithi098 ceph-mon[29473]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-12T03:34:13.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:13 smithi153 ceph-mon[25643]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:34:13.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:13 smithi098 ceph-mon[24773]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:34:13.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:13 smithi098 ceph-mon[29473]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-12T03:34:14.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:14 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:34:14.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:14 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:14 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:34:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:14 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:14.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:14 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T03:34:14.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:14 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:15.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:15 smithi153 ceph-mon[25643]: Deploying daemon osd.7 on smithi153 2024-03-12T03:34:15.689 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:15 smithi153 ceph-mon[25643]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-12T03:34:15.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:15 smithi098 ceph-mon[24773]: Deploying daemon osd.7 on smithi153 2024-03-12T03:34:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:15 smithi098 ceph-mon[24773]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-12T03:34:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:15 smithi098 ceph-mon[29473]: Deploying daemon osd.7 on smithi153 2024-03-12T03:34:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:15 smithi098 ceph-mon[29473]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-12T03:34:17.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:17 smithi153 ceph-mon[25643]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-12T03:34:17.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:17 smithi098 ceph-mon[24773]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-12T03:34:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:17 smithi098 ceph-mon[29473]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-12T03:34:19.684 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:19 smithi153 ceph-mon[25643]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:19.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:19 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:19.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:19 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:19.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:19 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:19.685 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:19 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[24773]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:19.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[29473]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:19.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:19.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:19.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:19.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:19 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:21 smithi153 ceph-mon[25643]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:21.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:21 smithi098 ceph-mon[24773]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:21.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:21 smithi098 ceph-mon[29473]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:22.613 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 7 on host 'smithi153' 2024-03-12T03:34:22.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:22.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:22.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:22.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:34:22.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:34:22.912 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:22.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:22.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:22.913 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:22 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:22 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:23.202 DEBUG:teuthology.orchestra.run.smithi153:osd.7> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.7.service 2024-03-12T03:34:23.204 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-12T03:34:23.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-12T03:34:24.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[29473]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[29473]: osdmap e52: 8 total, 7 up, 8 in 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[29473]: from='osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[29473]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[24773]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[24773]: osdmap e52: 8 total, 7 up, 8 in 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[24773]: from='osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:24 smithi098 ceph-mon[24773]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:24.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:24 smithi153 ceph-mon[25643]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T03:34:24.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:24 smithi153 ceph-mon[25643]: osdmap e52: 8 total, 7 up, 8 in 2024-03-12T03:34:24.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:24 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:24.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:24 smithi153 ceph-mon[25643]: from='osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:24.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:24 smithi153 ceph-mon[25643]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T03:34:24.367 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 03:34:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[37784]: 2024-03-12T03:34:24.084+0000 7f5f4f7f8700 -1 osd.7 0 waiting for initial osdmap 2024-03-12T03:34:24.367 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 03:34:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[37784]: 2024-03-12T03:34:24.092+0000 7f5f4b992700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T03:34:25.258 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:34:25.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[24773]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:34:25.258 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[24773]: osdmap e53: 8 total, 7 up, 8 in 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[24773]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[29473]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[29473]: osdmap e53: 8 total, 7 up, 8 in 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:25.259 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:25 smithi098 ceph-mon[29473]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:25.359 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:25 smithi153 ceph-mon[25643]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-03-12T03:34:25.360 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:25 smithi153 ceph-mon[25643]: osdmap e53: 8 total, 7 up, 8 in 2024-03-12T03:34:25.360 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:25 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:25.360 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:25 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:25.360 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:25 smithi153 ceph-mon[25643]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-12T03:34:25.891 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1710214465,"num_in_osds":8,"osd_in_since":1710214448,"num_remapped_pgs":1} 2024-03-12T03:34:25.892 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:34:26.118 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:26.239 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: purged_snaps scrub starts 2024-03-12T03:34:26.239 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: purged_snaps scrub ok 2024-03-12T03:34:26.239 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057] boot 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2565015952' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.240 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: purged_snaps scrub starts 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: purged_snaps scrub ok 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057] boot 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2565015952' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.241 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:26 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: purged_snaps scrub starts 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: purged_snaps scrub ok 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: osd.7 [v2:172.21.15.153:6824/1779830057,v1:172.21.15.153:6825/1779830057] boot 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: osdmap e54: 8 total, 8 up, 8 in 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2565015952' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:34:26.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:26.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:26 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi153 2024-03-12T03:34:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi153 to 4272M 2024-03-12T03:34:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[24773]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T03:34:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[24773]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi153 2024-03-12T03:34:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi153 to 4272M 2024-03-12T03:34:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[29473]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T03:34:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[29473]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:27 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:27 smithi153 ceph-mon[25643]: Detected new or changed devices on smithi153 2024-03-12T03:34:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:27 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi153 to 4272M 2024-03-12T03:34:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:27 smithi153 ceph-mon[25643]: osdmap e55: 8 total, 8 up, 8 in 2024-03-12T03:34:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:27 smithi153 ceph-mon[25643]: pgmap v165: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:27 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:28.115 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:34:28.116 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":57,"fsid":"ac0aaff8-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:29:34.097451+0000","modified":"2024-03-12T03:34:28.106320+0000","last_up_change":"2024-03-12T03:34:25.083863+0000","last_in_change":"2024-03-12T03:34:08.169592+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:32:32.649073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6803","nonce":1843576165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6805","nonce":1843576165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6809","nonce":1843576165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6807","nonce":1843576165}]},"public_addr":"172.21.15.98:6803/1843576165","cluster_addr":"172.21.15.98:6805/1843576165","heartbeat_back_addr":"172.21.15.98:6809/1843576165","heartbeat_front_addr":"172.21.15.98:6807/1843576165","state":["exists","up"]},{"osd":1,"uuid":"01a82066-05c4-4819-b27f-4aa3c611eaff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6811","nonce":1117288253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6813","nonce":1117288253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6817","nonce":1117288253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6815","nonce":1117288253}]},"public_addr":"172.21.15.98:6811/1117288253","cluster_addr":"172.21.15.98:6813/1117288253","heartbeat_back_addr":"172.21.15.98:6817/1117288253","heartbeat_front_addr":"172.21.15.98:6815/1117288253","state":["exists","up"]},{"osd":2,"uuid":"70b5963e-5214-4b47-a00f-d25a8237c8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6819","nonce":3197862193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6821","nonce":3197862193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6825","nonce":3197862193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6823","nonce":3197862193}]},"public_addr":"172.21.15.98:6819/3197862193","cluster_addr":"172.21.15.98:6821/3197862193","heartbeat_back_addr":"172.21.15.98:6825/3197862193","heartbeat_front_addr":"172.21.15.98:6823/3197862193","state":["exists","up"]},{"osd":3,"uuid":"c87f9327-c5f5-49b3-9505-2287534ab2b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6827","nonce":3388374879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6829","nonce":3388374879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6833","nonce":3388374879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6831","nonce":3388374879}]},"public_addr":"172.21.15.98:6827/3388374879","cluster_addr":"172.21.15.98:6829/3388374879","heartbeat_back_addr":"172.21.15.98:6833/3388374879","heartbeat_front_addr":"172.21.15.98:6831/3388374879","state":["exists","up"]},{"osd":4,"uuid":"73791fe8-e473-4d42-ad03-c32c85bc5738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6801","nonce":1148111885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6803","nonce":1148111885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6807","nonce":1148111885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6805","nonce":1148111885}]},"public_addr":"172.21.15.153:6801/1148111885","cluster_addr":"172.21.15.153:6803/1148111885","heartbeat_back_addr":"172.21.15.153:6807/1148111885","heartbeat_front_addr":"172.21.15.153:6805/1148111885","state":["exists","up"]},{"osd":5,"uuid":"7d75bfb5-af19-49cb-9c4e-9eae9ddc016e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6809","nonce":1227869641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6811","nonce":1227869641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6815","nonce":1227869641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6813","nonce":1227869641}]},"public_addr":"172.21.15.153:6809/1227869641","cluster_addr":"172.21.15.153:6811/1227869641","heartbeat_back_addr":"172.21.15.153:6815/1227869641","heartbeat_front_addr":"172.21.15.153:6813/1227869641","state":["exists","up"]},{"osd":6,"uuid":"c82a99fe-44eb-4019-ac81-355318f668ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6817","nonce":287724115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6819","nonce":287724115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6823","nonce":287724115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6821","nonce":287724115}]},"public_addr":"172.21.15.153:6817/287724115","cluster_addr":"172.21.15.153:6819/287724115","heartbeat_back_addr":"172.21.15.153:6823/287724115","heartbeat_front_addr":"172.21.15.153:6821/287724115","state":["exists","up"]},{"osd":7,"uuid":"abaf358b-9e89-468f-a1d5-3ed04d8ad102","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6825","nonce":1779830057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6827","nonce":1779830057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6831","nonce":1779830057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6829","nonce":1779830057}]},"public_addr":"172.21.15.153:6825/1779830057","cluster_addr":"172.21.15.153:6827/1779830057","heartbeat_back_addr":"172.21.15.153:6831/1779830057","heartbeat_front_addr":"172.21.15.153:6829/1779830057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:31:43.545327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:06.975450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:30.212782+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:53.997331+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:15.249232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:37.451734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:34:00.218003+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:34:23.435875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:6801/1041454473":"2024-03-13T03:30:26.275958+0000","172.21.15.98:6800/1041454473":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/4097120357":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/3533164705":"2024-03-13T03:30:04.233799+0000","172.21.15.98:0/1016008519":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/3827885316":"2024-03-13T03:30:04.233799+0000","172.21.15.98:0/3603574026":"2024-03-13T03:30:04.233799+0000","172.21.15.98:6800/2120159776":"2024-03-13T03:30:04.233799+0000","172.21.15.98:0/3824965392":"2024-03-13T03:30:26.275958+0000","172.21.15.98:6801/2120159776":"2024-03-13T03:30:04.233799+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:34:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:28 smithi153 ceph-mon[25643]: osdmap e56: 8 total, 8 up, 8 in 2024-03-12T03:34:28.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:28 smithi098 ceph-mon[24773]: osdmap e56: 8 total, 8 up, 8 in 2024-03-12T03:34:28.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:28 smithi098 ceph-mon[29473]: osdmap e56: 8 total, 8 up, 8 in 2024-03-12T03:34:28.898 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-12T03:32:32.649073+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-12T03:34:28.898 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-12T03:34:29.112 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:29.234 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:29 smithi098 ceph-mon[24773]: osdmap e57: 8 total, 8 up, 8 in 2024-03-12T03:34:29.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:29 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1572520250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:34:29.235 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:29 smithi098 ceph-mon[24773]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:29.235 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:29 smithi098 ceph-mon[29473]: osdmap e57: 8 total, 8 up, 8 in 2024-03-12T03:34:29.235 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:29 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1572520250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:34:29.235 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:29 smithi098 ceph-mon[29473]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:29.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:29 smithi153 ceph-mon[25643]: osdmap e57: 8 total, 8 up, 8 in 2024-03-12T03:34:29.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:29 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1572520250' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:34:29.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:29 smithi153 ceph-mon[25643]: pgmap v168: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:30.264 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:30 smithi098 ceph-mon[24773]: osdmap e58: 8 total, 8 up, 8 in 2024-03-12T03:34:30.264 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:30 smithi098 ceph-mon[29473]: osdmap e58: 8 total, 8 up, 8 in 2024-03-12T03:34:30.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:30 smithi153 ceph-mon[25643]: osdmap e58: 8 total, 8 up, 8 in 2024-03-12T03:34:30.945 INFO:teuthology.orchestra.run.smithi098.stdout:pg_num: 1 2024-03-12T03:34:31.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:31 smithi098 ceph-mon[24773]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:31.304 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:31 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1997578369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:34:31.304 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:31 smithi098 ceph-mon[29473]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:31.304 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:31 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1997578369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:34:31.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:31 smithi153 ceph-mon[25643]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:31.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:31 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1997578369' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-12T03:34:31.695 INFO:tasks.cephadm:Adding prometheus.a on smithi153 2024-03-12T03:34:31.695 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi153=a' 2024-03-12T03:34:33.539 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled prometheus update... 2024-03-12T03:34:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:33 smithi153 ceph-mon[25643]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-12T03:34:33.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:33 smithi098 ceph-mon[24773]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-12T03:34:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:33 smithi098 ceph-mon[29473]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-12T03:34:34.114 DEBUG:teuthology.orchestra.run.smithi153:prometheus.a> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-12T03:34:34.116 INFO:tasks.cephadm:Adding node-exporter.a on smithi098 2024-03-12T03:34:34.117 INFO:tasks.cephadm:Adding node-exporter.b on smithi153 2024-03-12T03:34:34.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi098=a;smithi153=b' 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: from='client.34307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: Saving service prometheus spec with placement smithi153=a;count:1 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:34:34.622 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mon[25643]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: from='client.34307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: Saving service prometheus spec with placement smithi153=a;count:1 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:34:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[24773]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: from='client.34307 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: Saving service prometheus spec with placement smithi153=a;count:1 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-12T03:34:34.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mon[29473]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-03-12T03:34:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setuser ceph since I am not root 2024-03-12T03:34:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setgroup ceph since I am not root 2024-03-12T03:34:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:34 smithi098 ceph-mgr[25009]: -- 172.21.15.98:0/819763477 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56284f52c340 con 0x56284f504c00 2024-03-12T03:34:34.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ignoring --setuser ceph since I am not root 2024-03-12T03:34:34.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ignoring --setgroup ceph since I am not root 2024-03-12T03:34:34.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:34 smithi153 ceph-mgr[26825]: -- 172.21.15.153:0/1868171719 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559702266340 con 0x55970223e800 2024-03-12T03:34:35.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:35.103+0000 7f985f3f2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:34:35.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:35.195+0000 7f985f3f2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:34:35.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:35.086+0000 7fca0514f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:34:35.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:35.174+0000 7fca0514f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:34:35.728 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:35 smithi153 ceph-mon[25643]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:34:35.728 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:35 smithi153 ceph-mon[25643]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:34:35.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-mon[24773]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:34:35.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-mon[24773]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:34:35.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-mon[29473]: from='mgr.14150 172.21.15.98:0/2545264592' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-12T03:34:35.850 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-mon[29473]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-12T03:34:35.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:35.759+0000 7fca0514f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:34:35.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:35.847+0000 7fca0514f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:34:36.007 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:35.824+0000 7f985f3f2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:34:36.008 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:35.916+0000 7f985f3f2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:34:36.008 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:36.004+0000 7f985f3f2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:34:36.159 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:35.929+0000 7fca0514f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:34:36.160 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:36 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:36.077+0000 7fca0514f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:34:36.160 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:36 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:36.157+0000 7fca0514f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:34:36.352 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:36.158+0000 7f985f3f2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:34:36.352 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:36.238+0000 7f985f3f2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:34:36.352 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:36.348+0000 7f985f3f2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:34:36.513 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:36 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:36.266+0000 7fca0514f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:34:36.513 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:36 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:36.350+0000 7fca0514f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:34:36.514 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:36 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:36.511+0000 7fca0514f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:34:36.685 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:36.432+0000 7f985f3f2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:34:36.685 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:36.596+0000 7f985f3f2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:34:36.685 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:36.682+0000 7f985f3f2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:34:36.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:36 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:36.595+0000 7fca0514f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:34:37.925 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:37.604+0000 7fca0514f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:34:37.925 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:37.734+0000 7fca0514f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:34:37.925 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:37.922+0000 7fca0514f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:34:38.024 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:37 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:37.704+0000 7f985f3f2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:34:38.024 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:37 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:37.832+0000 7f985f3f2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:34:38.024 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:38.021+0000 7f985f3f2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:34:38.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:38.009+0000 7fca0514f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:34:38.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:38.144+0000 7fca0514f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:34:38.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:38.109+0000 7f985f3f2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:34:38.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:38.245+0000 7f985f3f2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:34:38.769 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:38.461+0000 7fca0514f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:34:38.769 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:38.682+0000 7fca0514f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:34:38.872 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:38.565+0000 7f985f3f2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:34:38.872 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:38.785+0000 7f985f3f2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:34:38.872 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:38.869+0000 7f985f3f2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:34:39.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:38.767+0000 7fca0514f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:34:39.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:38.936+0000 7fca0514f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:34:39.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:39.037+0000 7f985f3f2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:34:40.139 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:39.844+0000 7fca0514f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:34:40.243 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:39.948+0000 7f985f3f2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:34:40.243 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:40.240+0000 7f985f3f2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:34:40.455 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:40.136+0000 7fca0514f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:34:40.455 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:40.232+0000 7fca0514f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:34:40.455 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:40.380+0000 7fca0514f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:34:40.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-mon[25643]: Active manager daemon y restarted 2024-03-12T03:34:40.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-mon[25643]: Activating manager daemon y 2024-03-12T03:34:40.505 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-mon[25643]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:34:40.506 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:40.334+0000 7f985f3f2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:34:40.506 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:40.487+0000 7f985f3f2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:34:40.506 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: [12/Mar/2024:03:34:40] ENGINE Bus STARTING 2024-03-12T03:34:40.783 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:34:40] ENGINE Bus STARTING 2024-03-12T03:34:40.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-mon[24773]: Active manager daemon y restarted 2024-03-12T03:34:40.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-mon[24773]: Activating manager daemon y 2024-03-12T03:34:40.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-mon[24773]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:34:40.783 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-mon[29473]: Active manager daemon y restarted 2024-03-12T03:34:40.783 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-mon[29473]: Activating manager daemon y 2024-03-12T03:34:40.783 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-mon[29473]: osdmap e59: 8 total, 8 up, 8 in 2024-03-12T03:34:40.864 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: CherryPy Checker: 2024-03-12T03:34:40.864 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: The Application mounted at '' has an empty config. 2024-03-12T03:34:40.864 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: 2024-03-12T03:34:40.864 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: [12/Mar/2024:03:34:40] ENGINE Serving on http://:::9283 2024-03-12T03:34:40.864 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:34:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: [12/Mar/2024:03:34:40] ENGINE Bus STARTED 2024-03-12T03:34:41.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: CherryPy Checker: 2024-03-12T03:34:41.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: The Application mounted at '' has an empty config. 2024-03-12T03:34:41.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:34:41.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:34:40] ENGINE Serving on http://:::9283 2024-03-12T03:34:41.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:34:40] ENGINE Bus STARTED 2024-03-12T03:34:41.452 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:34:41] ENGINE Bus STARTING 2024-03-12T03:34:41.453 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:34:41] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:34:41.453 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:34:41] ENGINE Bus STARTED 2024-03-12T03:34:41.484 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled node-exporter update... 2024-03-12T03:34:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: mgrmap e17: y(active, starting, since 0.0562986s), standbys: x 2024-03-12T03:34:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:34:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:34:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:34:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:34:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:34:41.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: Manager daemon y is now available 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: Standby manager daemon x restarted 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: Standby manager daemon x started 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:41.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:41.827 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:41.827 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:41.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: mgrmap e17: y(active, starting, since 0.0562986s), standbys: x 2024-03-12T03:34:41.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:34:41.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:34:41.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:34:41.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: Manager daemon y is now available 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:41.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: Standby manager daemon x restarted 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: Standby manager daemon x started 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:41.829 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:41.830 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: mgrmap e17: y(active, starting, since 0.0562986s), standbys: x 2024-03-12T03:34:41.830 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:34:41.830 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:34:41.830 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:34:41.830 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:34:41.830 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:34:41.830 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: Manager daemon y is now available 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:41.831 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: Standby manager daemon x restarted 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: Standby manager daemon x started 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/4043139785' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:34:41.832 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:42.454 DEBUG:teuthology.orchestra.run.smithi098:node-exporter.a> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-12T03:34:42.456 DEBUG:teuthology.orchestra.run.smithi153:node-exporter.b> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-12T03:34:42.459 INFO:tasks.cephadm:Adding alertmanager.a on smithi098 2024-03-12T03:34:42.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi098=a' 2024-03-12T03:34:42.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: [12/Mar/2024:03:34:41] ENGINE Bus STARTING 2024-03-12T03:34:42.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: [12/Mar/2024:03:34:41] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: [12/Mar/2024:03:34:41] ENGINE Bus STARTED 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: mgrmap e18: y(active, since 1.07712s), standbys: x 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi098=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: Saving service node-exporter spec with placement smithi098=a;smithi153=b;count:2 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: [12/Mar/2024:03:34:41] ENGINE Bus STARTING 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: [12/Mar/2024:03:34:41] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: [12/Mar/2024:03:34:41] ENGINE Bus STARTED 2024-03-12T03:34:42.643 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: mgrmap e18: y(active, since 1.07712s), standbys: x 2024-03-12T03:34:42.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi098=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:42.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:42.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: Saving service node-exporter spec with placement smithi098=a;smithi153=b;count:2 2024-03-12T03:34:42.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:42.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:42 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: [12/Mar/2024:03:34:41] ENGINE Bus STARTING 2024-03-12T03:34:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: [12/Mar/2024:03:34:41] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:34:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: [12/Mar/2024:03:34:41] ENGINE Bus STARTED 2024-03-12T03:34:42.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: mgrmap e18: y(active, since 1.07712s), standbys: x 2024-03-12T03:34:42.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi098=a;smithi153=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:42.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:42.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: Saving service node-exporter spec with placement smithi098=a;smithi153=b;count:2 2024-03-12T03:34:42.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:42.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:42 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:43 smithi153 ceph-mon[25643]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:43.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:43 smithi153 ceph-mon[25643]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-12T03:34:43.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[24773]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[24773]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[29473]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[29473]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-12T03:34:43.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:43.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:44.527 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled alertmanager update... 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:34:44.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:44.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:44.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:44 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:44.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:44 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:34:44.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:44 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:44.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:44 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:34:44.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:44 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:34:44.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:44 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:44.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:44 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:45.105 DEBUG:teuthology.orchestra.run.smithi098:alertmanager.a> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-12T03:34:45.108 INFO:tasks.cephadm:Adding grafana.a on smithi153 2024-03-12T03:34:45.108 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi153=a' 2024-03-12T03:34:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:45 smithi153 ceph-mon[25643]: Deploying daemon prometheus.a on smithi153 2024-03-12T03:34:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:45 smithi153 ceph-mon[25643]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:45 smithi153 ceph-mon[25643]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:45 smithi153 ceph-mon[25643]: Saving service alertmanager spec with placement smithi098=a;count:1 2024-03-12T03:34:45.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:45 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[24773]: Deploying daemon prometheus.a on smithi153 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[24773]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[24773]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[24773]: Saving service alertmanager spec with placement smithi098=a;count:1 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[29473]: Deploying daemon prometheus.a on smithi153 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[29473]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[29473]: from='client.34352 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi098=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:45.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[29473]: Saving service alertmanager spec with placement smithi098=a;count:1 2024-03-12T03:34:45.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:45 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:47 smithi098 ceph-mon[24773]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:47 smithi098 ceph-mon[29473]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:47.850 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:47 smithi153 ceph-mon[25643]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:48.003 INFO:teuthology.orchestra.run.smithi153.stdout:Scheduled grafana update... 2024-03-12T03:34:48.909 DEBUG:teuthology.orchestra.run.smithi153:grafana.a> sudo journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service 2024-03-12T03:34:48.911 INFO:tasks.cephadm:Setting up client nodes... 2024-03-12T03:34:48.914 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:34:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:48 smithi153 ceph-mon[25643]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:48 smithi153 ceph-mon[25643]: Saving service grafana spec with placement smithi153=a;count:1 2024-03-12T03:34:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:48 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:49.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:48 smithi153 ceph-mon[25643]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:49.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:48 smithi098 ceph-mon[24773]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:49.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:48 smithi098 ceph-mon[24773]: Saving service grafana spec with placement smithi153=a;count:1 2024-03-12T03:34:49.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:48 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:49.228 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:48 smithi098 ceph-mon[24773]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:49.229 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:49 smithi098 ceph-mon[29473]: from='client.34358 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi153=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:34:49.229 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:49 smithi098 ceph-mon[29473]: Saving service grafana spec with placement smithi153=a;count:1 2024-03-12T03:34:49.229 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:49 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:49.229 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:49 smithi098 ceph-mon[29473]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:50.936 INFO:teuthology.orchestra.run.smithi098.stdout:[client.0] 2024-03-12T03:34:50.936 INFO:teuthology.orchestra.run.smithi098.stdout: key = AQBaze9lCVV7NxAAbRQgBkOsl8xw+QnM55/o2g== 2024-03-12T03:34:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:51 smithi098 ceph-mon[24773]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:51 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2115648512' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:51 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2115648512' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:34:51.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:51 smithi098 ceph-mon[29473]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:51.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:51 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2115648512' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:51.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:51 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2115648512' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:34:51.614 DEBUG:teuthology.orchestra.run.smithi098:> set -ex 2024-03-12T03:34:51.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-12T03:34:51.614 DEBUG:teuthology.orchestra.run.smithi098:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-12T03:34:51.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-12T03:34:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:51 smithi153 ceph-mon[25643]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:51 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2115648512' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:51 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2115648512' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:34:53.802 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:53 smithi153 ceph-mon[25643]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:53.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:53 smithi098 ceph-mon[24773]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:53 smithi098 ceph-mon[29473]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:54.844 INFO:teuthology.orchestra.run.smithi153.stdout:[client.1] 2024-03-12T03:34:54.844 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBeze9lDVW+MRAAfy96k7QobatGdLS7VY3a4w== 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[24773]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[24773]: from='client.? 172.21.15.153:0/1835642990' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[29473]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[29473]: from='client.? 172.21.15.153:0/1835642990' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:55 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:34:55.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:55 smithi153 ceph-mon[25643]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:55.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:55 smithi153 ceph-mon[25643]: from='client.? 172.21.15.153:0/1835642990' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:55.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:55 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-12T03:34:55.876 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:55 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-12T03:34:56.500 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-03-12T03:34:56.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-12T03:34:56.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-12T03:34:56.547 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-12T03:34:56.547 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-12T03:34:56.547 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-12T03:34:56.767 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:57.780 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:57 smithi098 ceph-mon[24773]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:57.780 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:57 smithi098 ceph-mon[29473]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:57 smithi153 ceph-mon[25643]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:58.444 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:34:58.691 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 systemd[1]: Starting Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:34:58.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 podman[40762]: 2024-03-12 03:34:58.689638013 +0000 UTC m=+0.116441181 container create 6086ef920aa3383c3d375b4b2b5b64198cd759e26e5ecd21dc3d518263311639 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:58.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 podman[40762]: 2024-03-12 03:34:58.593851602 +0000 UTC m=+0.020654760 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-12T03:34:58.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 podman[40762]: 2024-03-12 03:34:58.908489894 +0000 UTC m=+0.335293068 container init 6086ef920aa3383c3d375b4b2b5b64198cd759e26e5ecd21dc3d518263311639 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 podman[40762]: 2024-03-12 03:34:58.911026757 +0000 UTC m=+0.337829935 container start 6086ef920aa3383c3d375b4b2b5b64198cd759e26e5ecd21dc3d518263311639 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.934Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.934Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.934Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.934Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi153 (none))" 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.934Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.934Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.935Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.936Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.937Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.939Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.939Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.15µs 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.939Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:34:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.939Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-12T03:34:58.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.939Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=21.843µs wal_replay_duration=178.087µs total_replay_duration=215.569µs 2024-03-12T03:34:58.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.940Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:34:58.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.940Z caller=main.go:947 level=info msg="TSDB started" 2024-03-12T03:34:58.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.940Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:34:59.192 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.955Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.121607ms db_storage=898ns remote_storage=1.577µs web_handler=455ns query_engine=718ns scrape=629.945µs scrape_sd=23.961µs notify=1.072µs notify_sd=2.121µs rules=14.22193ms 2024-03-12T03:34:59.192 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:34:58.955Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-12T03:34:59.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 bash[40762]: 6086ef920aa3383c3d375b4b2b5b64198cd759e26e5ecd21dc3d518263311639 2024-03-12T03:34:59.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:34:58 smithi153 systemd[1]: Started Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:34:59.194 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":19,"active_gid":24343,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6800","nonce":1273287586},{"type":"v1","addr":"172.21.15.98:6801","nonce":1273287586}]},"active_addr":"172.21.15.98:6801/1273287586","active_change":"2024-03-12T03:34:40.394619+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34313,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.98:8443/","prometheus":"http://172.21.15.98:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":1958076942}]},{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":2704764386}]},{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":4044504336}]},{"addrvec":[{"type":"v2","addr":"172.21.15.98:0","nonce":3833154681}]}]}} 2024-03-12T03:34:59.197 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-12T03:34:59.197 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-12T03:34:59.197 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:34:59.426 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:34:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:59 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/203528267' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:34:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:59 smithi153 ceph-mon[25643]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:59.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:34:59 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:59.802 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:59 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/203528267' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:34:59.802 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:59 smithi098 ceph-mon[24773]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:59.802 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:34:59 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:34:59.802 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:59 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/203528267' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:34:59.802 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:59 smithi098 ceph-mon[29473]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:34:59.802 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:34:59 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:00.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:00 smithi153 ceph-mon[25643]: Deploying daemon node-exporter.a on smithi098 2024-03-12T03:35:00.754 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:00 smithi098 ceph-mon[24773]: Deploying daemon node-exporter.a on smithi098 2024-03-12T03:35:00.754 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:00 smithi098 ceph-mon[29473]: Deploying daemon node-exporter.a on smithi098 2024-03-12T03:35:01.461 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:35:01.461 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":59,"fsid":"ac0aaff8-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:29:34.097451+0000","modified":"2024-03-12T03:34:40.394298+0000","last_up_change":"2024-03-12T03:34:25.083863+0000","last_in_change":"2024-03-12T03:34:08.169592+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:32:32.649073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6803","nonce":1843576165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6805","nonce":1843576165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6809","nonce":1843576165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6807","nonce":1843576165}]},"public_addr":"172.21.15.98:6803/1843576165","cluster_addr":"172.21.15.98:6805/1843576165","heartbeat_back_addr":"172.21.15.98:6809/1843576165","heartbeat_front_addr":"172.21.15.98:6807/1843576165","state":["exists","up"]},{"osd":1,"uuid":"01a82066-05c4-4819-b27f-4aa3c611eaff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6811","nonce":1117288253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6813","nonce":1117288253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6817","nonce":1117288253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6815","nonce":1117288253}]},"public_addr":"172.21.15.98:6811/1117288253","cluster_addr":"172.21.15.98:6813/1117288253","heartbeat_back_addr":"172.21.15.98:6817/1117288253","heartbeat_front_addr":"172.21.15.98:6815/1117288253","state":["exists","up"]},{"osd":2,"uuid":"70b5963e-5214-4b47-a00f-d25a8237c8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6819","nonce":3197862193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6821","nonce":3197862193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6825","nonce":3197862193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6823","nonce":3197862193}]},"public_addr":"172.21.15.98:6819/3197862193","cluster_addr":"172.21.15.98:6821/3197862193","heartbeat_back_addr":"172.21.15.98:6825/3197862193","heartbeat_front_addr":"172.21.15.98:6823/3197862193","state":["exists","up"]},{"osd":3,"uuid":"c87f9327-c5f5-49b3-9505-2287534ab2b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6827","nonce":3388374879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6829","nonce":3388374879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6833","nonce":3388374879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6831","nonce":3388374879}]},"public_addr":"172.21.15.98:6827/3388374879","cluster_addr":"172.21.15.98:6829/3388374879","heartbeat_back_addr":"172.21.15.98:6833/3388374879","heartbeat_front_addr":"172.21.15.98:6831/3388374879","state":["exists","up"]},{"osd":4,"uuid":"73791fe8-e473-4d42-ad03-c32c85bc5738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6801","nonce":1148111885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6803","nonce":1148111885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6807","nonce":1148111885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6805","nonce":1148111885}]},"public_addr":"172.21.15.153:6801/1148111885","cluster_addr":"172.21.15.153:6803/1148111885","heartbeat_back_addr":"172.21.15.153:6807/1148111885","heartbeat_front_addr":"172.21.15.153:6805/1148111885","state":["exists","up"]},{"osd":5,"uuid":"7d75bfb5-af19-49cb-9c4e-9eae9ddc016e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6809","nonce":1227869641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6811","nonce":1227869641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6815","nonce":1227869641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6813","nonce":1227869641}]},"public_addr":"172.21.15.153:6809/1227869641","cluster_addr":"172.21.15.153:6811/1227869641","heartbeat_back_addr":"172.21.15.153:6815/1227869641","heartbeat_front_addr":"172.21.15.153:6813/1227869641","state":["exists","up"]},{"osd":6,"uuid":"c82a99fe-44eb-4019-ac81-355318f668ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6817","nonce":287724115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6819","nonce":287724115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6823","nonce":287724115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6821","nonce":287724115}]},"public_addr":"172.21.15.153:6817/287724115","cluster_addr":"172.21.15.153:6819/287724115","heartbeat_back_addr":"172.21.15.153:6823/287724115","heartbeat_front_addr":"172.21.15.153:6821/287724115","state":["exists","up"]},{"osd":7,"uuid":"abaf358b-9e89-468f-a1d5-3ed04d8ad102","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6825","nonce":1779830057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6827","nonce":1779830057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6831","nonce":1779830057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6829","nonce":1779830057}]},"public_addr":"172.21.15.153:6825/1779830057","cluster_addr":"172.21.15.153:6827/1779830057","heartbeat_back_addr":"172.21.15.153:6831/1779830057","heartbeat_front_addr":"172.21.15.153:6829/1779830057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:31:43.545327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:06.975450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:30.212782+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:53.997331+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:15.249232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:37.451734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:34:00.218003+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:34:23.435875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3339782884":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/1032522663":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/3485535046":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/3469331030":"2024-03-13T03:34:40.394268+0000","172.21.15.98:6801/1041454473":"2024-03-13T03:30:26.275958+0000","172.21.15.98:6800/85873921":"2024-03-13T03:34:40.394268+0000","172.21.15.98:6800/1041454473":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/4097120357":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/3533164705":"2024-03-13T03:30:04.233799+0000","172.21.15.98:0/1016008519":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/3827885316":"2024-03-13T03:30:04.233799+0000","172.21.15.98:0/3603574026":"2024-03-13T03:30:04.233799+0000","172.21.15.98:6800/2120159776":"2024-03-13T03:30:04.233799+0000","172.21.15.98:6801/85873921":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/3824965392":"2024-03-13T03:30:26.275958+0000","172.21.15.98:6801/2120159776":"2024-03-13T03:30:04.233799+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:01 smithi098 ceph-mon[24773]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:01 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:01 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2428723506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:35:01.748 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:01 smithi098 ceph-mon[29473]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:01.749 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:01 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:01.749 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:01 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2428723506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:35:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:01 smithi153 ceph-mon[25643]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:01.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:01 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:01.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:01 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2428723506' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:35:02.128 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-12T03:35:02.128 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-12T03:35:02.784 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:03.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:03 smithi098 ceph-mon[24773]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:03 smithi098 ceph-mon[29473]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:03 smithi153 ceph-mon[25643]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:04.262 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 podman[44359]: 2024-03-12 03:35:04.057800192 +0000 UTC m=+2.806448901 container create fd51a335b42c7cc1f8b575ea8ce74da6b45af3e997d091cf2d6ee7b06ab7ce90 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:04.262 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 podman[44359]: 2024-03-12 03:35:04.259881212 +0000 UTC m=+3.008529936 container init fd51a335b42c7cc1f8b575ea8ce74da6b45af3e997d091cf2d6ee7b06ab7ce90 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 podman[44359]: 2024-03-12 03:35:04.263205038 +0000 UTC m=+3.011853749 container start fd51a335b42c7cc1f8b575ea8ce74da6b45af3e997d091cf2d6ee7b06ab7ce90 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.267Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.267Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.268Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.268Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=arp 2024-03-12T03:35:04.557 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=edac 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-12T03:35:04.558 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-12T03:35:04.559 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=os 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=stat 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-12T03:35:04.560 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=time 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=uname 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[44520]: ts=2024-03-12T03:35:04.269Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 bash[44359]: fd51a335b42c7cc1f8b575ea8ce74da6b45af3e997d091cf2d6ee7b06ab7ce90 2024-03-12T03:35:04.561 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:35:04 smithi098 systemd[1]: Started Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:35:05.442 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:05 smithi153 bash[41006]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-12T03:35:05.474 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:35:05.474 INFO:teuthology.orchestra.run.smithi098.stdout:{"epoch":59,"fsid":"ac0aaff8-e020-11ee-95c8-87774f69a715","created":"2024-03-12T03:29:34.097451+0000","modified":"2024-03-12T03:34:40.394298+0000","last_up_change":"2024-03-12T03:34:25.083863+0000","last_in_change":"2024-03-12T03:34:08.169592+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-12T03:32:32.649073+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"79f13fc6-6d2f-4a1e-8ed3-8c74bc306b7b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6802","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6803","nonce":1843576165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6804","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6805","nonce":1843576165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6808","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6809","nonce":1843576165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6806","nonce":1843576165},{"type":"v1","addr":"172.21.15.98:6807","nonce":1843576165}]},"public_addr":"172.21.15.98:6803/1843576165","cluster_addr":"172.21.15.98:6805/1843576165","heartbeat_back_addr":"172.21.15.98:6809/1843576165","heartbeat_front_addr":"172.21.15.98:6807/1843576165","state":["exists","up"]},{"osd":1,"uuid":"01a82066-05c4-4819-b27f-4aa3c611eaff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6810","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6811","nonce":1117288253}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6812","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6813","nonce":1117288253}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6816","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6817","nonce":1117288253}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6814","nonce":1117288253},{"type":"v1","addr":"172.21.15.98:6815","nonce":1117288253}]},"public_addr":"172.21.15.98:6811/1117288253","cluster_addr":"172.21.15.98:6813/1117288253","heartbeat_back_addr":"172.21.15.98:6817/1117288253","heartbeat_front_addr":"172.21.15.98:6815/1117288253","state":["exists","up"]},{"osd":2,"uuid":"70b5963e-5214-4b47-a00f-d25a8237c8a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6818","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6819","nonce":3197862193}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6820","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6821","nonce":3197862193}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6824","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6825","nonce":3197862193}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6822","nonce":3197862193},{"type":"v1","addr":"172.21.15.98:6823","nonce":3197862193}]},"public_addr":"172.21.15.98:6819/3197862193","cluster_addr":"172.21.15.98:6821/3197862193","heartbeat_back_addr":"172.21.15.98:6825/3197862193","heartbeat_front_addr":"172.21.15.98:6823/3197862193","state":["exists","up"]},{"osd":3,"uuid":"c87f9327-c5f5-49b3-9505-2287534ab2b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6826","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6827","nonce":3388374879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6828","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6829","nonce":3388374879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6832","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6833","nonce":3388374879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.98:6830","nonce":3388374879},{"type":"v1","addr":"172.21.15.98:6831","nonce":3388374879}]},"public_addr":"172.21.15.98:6827/3388374879","cluster_addr":"172.21.15.98:6829/3388374879","heartbeat_back_addr":"172.21.15.98:6833/3388374879","heartbeat_front_addr":"172.21.15.98:6831/3388374879","state":["exists","up"]},{"osd":4,"uuid":"73791fe8-e473-4d42-ad03-c32c85bc5738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6801","nonce":1148111885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6803","nonce":1148111885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6807","nonce":1148111885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1148111885},{"type":"v1","addr":"172.21.15.153:6805","nonce":1148111885}]},"public_addr":"172.21.15.153:6801/1148111885","cluster_addr":"172.21.15.153:6803/1148111885","heartbeat_back_addr":"172.21.15.153:6807/1148111885","heartbeat_front_addr":"172.21.15.153:6805/1148111885","state":["exists","up"]},{"osd":5,"uuid":"7d75bfb5-af19-49cb-9c4e-9eae9ddc016e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6809","nonce":1227869641}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6811","nonce":1227869641}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6815","nonce":1227869641}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":1227869641},{"type":"v1","addr":"172.21.15.153:6813","nonce":1227869641}]},"public_addr":"172.21.15.153:6809/1227869641","cluster_addr":"172.21.15.153:6811/1227869641","heartbeat_back_addr":"172.21.15.153:6815/1227869641","heartbeat_front_addr":"172.21.15.153:6813/1227869641","state":["exists","up"]},{"osd":6,"uuid":"c82a99fe-44eb-4019-ac81-355318f668ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6817","nonce":287724115}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6819","nonce":287724115}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6823","nonce":287724115}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":287724115},{"type":"v1","addr":"172.21.15.153:6821","nonce":287724115}]},"public_addr":"172.21.15.153:6817/287724115","cluster_addr":"172.21.15.153:6819/287724115","heartbeat_back_addr":"172.21.15.153:6823/287724115","heartbeat_front_addr":"172.21.15.153:6821/287724115","state":["exists","up"]},{"osd":7,"uuid":"abaf358b-9e89-468f-a1d5-3ed04d8ad102","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6825","nonce":1779830057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6826","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6827","nonce":1779830057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6830","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6831","nonce":1779830057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6828","nonce":1779830057},{"type":"v1","addr":"172.21.15.153:6829","nonce":1779830057}]},"public_addr":"172.21.15.153:6825/1779830057","cluster_addr":"172.21.15.153:6827/1779830057","heartbeat_back_addr":"172.21.15.153:6831/1779830057","heartbeat_front_addr":"172.21.15.153:6829/1779830057","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:31:43.545327+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:06.975450+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:30.212782+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:32:53.997331+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:15.249232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:33:37.451734+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:34:00.218003+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-12T03:34:23.435875+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.98:0/3339782884":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/1032522663":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/3485535046":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/3469331030":"2024-03-13T03:34:40.394268+0000","172.21.15.98:6801/1041454473":"2024-03-13T03:30:26.275958+0000","172.21.15.98:6800/85873921":"2024-03-13T03:34:40.394268+0000","172.21.15.98:6800/1041454473":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/4097120357":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/3533164705":"2024-03-13T03:30:04.233799+0000","172.21.15.98:0/1016008519":"2024-03-13T03:30:26.275958+0000","172.21.15.98:0/3827885316":"2024-03-13T03:30:04.233799+0000","172.21.15.98:0/3603574026":"2024-03-13T03:30:04.233799+0000","172.21.15.98:6800/2120159776":"2024-03-13T03:30:04.233799+0000","172.21.15.98:6801/85873921":"2024-03-13T03:34:40.394268+0000","172.21.15.98:0/3824965392":"2024-03-13T03:30:26.275958+0000","172.21.15.98:6801/2120159776":"2024-03-13T03:30:04.233799+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-12T03:35:05.771 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:05 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:05.771 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:05 smithi098 ceph-mon[29473]: Deploying daemon node-exporter.b on smithi153 2024-03-12T03:35:05.771 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:05 smithi098 ceph-mon[29473]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:05.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:05 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:05.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:05 smithi098 ceph-mon[24773]: Deploying daemon node-exporter.b on smithi153 2024-03-12T03:35:05.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:05 smithi098 ceph-mon[24773]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:05 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:05 smithi153 ceph-mon[25643]: Deploying daemon node-exporter.b on smithi153 2024-03-12T03:35:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:05 smithi153 ceph-mon[25643]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:05.943 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:05 smithi153 bash[41006]: Getting image source signatures 2024-03-12T03:35:05.943 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:05 smithi153 bash[41006]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-12T03:35:05.943 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:05 smithi153 bash[41006]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-12T03:35:05.943 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:05 smithi153 bash[41006]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-12T03:35:06.137 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-12T03:35:06.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-12T03:35:06.138 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-12T03:35:06.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-12T03:35:06.139 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-12T03:35:06.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-12T03:35:06.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-12T03:35:06.140 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-12T03:35:06.449 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:06 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/524248439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:35:06.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:06 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/524248439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:35:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.549 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.550 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.551 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.557 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.560 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:06.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:06 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/524248439' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-12T03:35:07.192 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:06 smithi153 bash[41006]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-12T03:35:07.192 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 bash[41006]: Writing manifest to image destination 2024-03-12T03:35:07.621 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-mon[25643]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:07.622 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 podman[41006]: 2024-03-12 03:35:07.235746002 +0000 UTC m=+2.036522869 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-12T03:35:07.622 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 podman[41006]: 2024-03-12 03:35:07.325964109 +0000 UTC m=+2.126740957 container create 270f07cc8db829f7a928d4a750cd395bf4274a986e0452426ce472939643528c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:35:07.622 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 podman[41006]: 2024-03-12 03:35:07.544592083 +0000 UTC m=+2.345368940 container init 270f07cc8db829f7a928d4a750cd395bf4274a986e0452426ce472939643528c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:35:07.622 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 podman[41006]: 2024-03-12 03:35:07.547739478 +0000 UTC m=+2.348516334 container start 270f07cc8db829f7a928d4a750cd395bf4274a986e0452426ce472939643528c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.551Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.551Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=arp 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-12T03:35:07.623 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=edac 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.552Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-12T03:35:07.624 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=os 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-12T03:35:07.625 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=stat 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=time 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=uname 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-12T03:35:07.626 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-12T03:35:07.627 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[41059]: ts=2024-03-12T03:35:07.553Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-12T03:35:07.627 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 bash[41006]: 270f07cc8db829f7a928d4a750cd395bf4274a986e0452426ce472939643528c 2024-03-12T03:35:07.647 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:07 smithi098 ceph-mon[24773]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:07.648 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:07 smithi098 ceph-mon[29473]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:07.942 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:35:07 smithi153 systemd[1]: Started Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:35:08.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:08.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:08.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:08.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:08.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[24773]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:08.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:08.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:08.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:08.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:08.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:08 smithi098 ceph-mon[29473]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:09.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:08 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:09.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:08 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:35:09.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:08 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:35:09.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:08 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:35:09.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:08 smithi153 ceph-mon[25643]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:09.669 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:35:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:09] "GET /metrics HTTP/1.1" 200 191503 "" "Prometheus/2.33.4" 2024-03-12T03:35:09.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:35:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:35:11.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:11 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:11.230 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:11 smithi098 ceph-mon[24773]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:11.230 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:11 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:11.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:11 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:11.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:11 smithi098 ceph-mon[29473]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:11.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:11 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:11 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:11.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:11 smithi153 ceph-mon[25643]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:11.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:11 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:13.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:13 smithi098 ceph-mon[24773]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:13 smithi098 ceph-mon[29473]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:13 smithi153 ceph-mon[25643]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:15.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:15 smithi098 ceph-mon[24773]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:15.760 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:15 smithi098 ceph-mon[29473]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:15 smithi153 ceph-mon[25643]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:17.778 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:17 smithi153 ceph-mon[25643]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:17.789 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:17 smithi098 ceph-mon[24773]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:17.790 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:17 smithi098 ceph-mon[29473]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:18.278 INFO:teuthology.orchestra.run.smithi098.stdout:231928233996 2024-03-12T03:35:18.279 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-12T03:35:19.535 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:19 smithi098 ceph-mon[24773]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:19.535 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:19 smithi098 ceph-mon[29473]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:19 smithi153 ceph-mon[25643]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:19.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:35:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:35:19.804 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:35:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:19] "GET /metrics HTTP/1.1" 200 191484 "" "Prometheus/2.33.4" 2024-03-12T03:35:20.187 INFO:teuthology.orchestra.run.smithi098.stdout:163208757270 2024-03-12T03:35:20.187 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-12T03:35:20.789 INFO:teuthology.orchestra.run.smithi098.stdout:197568495634 2024-03-12T03:35:20.790 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-12T03:35:21.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:21 smithi098 ceph-mon[24773]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:21.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:21 smithi098 ceph-mon[29473]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:21 smithi153 ceph-mon[25643]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:21.988 INFO:teuthology.orchestra.run.smithi098.stdout:107374182430 2024-03-12T03:35:21.988 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-12T03:35:22.692 INFO:teuthology.orchestra.run.smithi098.stdout:55834574887 2024-03-12T03:35:22.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-12T03:35:23.631 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:23.659 INFO:teuthology.orchestra.run.smithi098.stdout:128849018905 2024-03-12T03:35:23.659 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-12T03:35:23.660 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:23.662 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:23.663 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:23.663 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:23 smithi098 ceph-mon[29473]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:23.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:23 smithi098 ceph-mon[24773]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:23 smithi153 ceph-mon[25643]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:24.095 INFO:teuthology.orchestra.run.smithi098.stdout:34359738413 2024-03-12T03:35:24.095 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-12T03:35:24.346 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:25.776 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:25 smithi098 ceph-mon[24773]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:25.777 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:25 smithi098 ceph-mon[29473]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:25 smithi153 ceph-mon[25643]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:27 smithi098 ceph-mon[24773]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:27 smithi098 ceph-mon[29473]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:27.903 INFO:teuthology.orchestra.run.smithi098.stdout:77309411365 2024-03-12T03:35:27.903 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-12T03:35:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:27 smithi153 ceph-mon[25643]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:29.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:29 smithi098 ceph-mon[24773]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:29.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:29 smithi098 ceph-mon[29473]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:29 smithi153 ceph-mon[25643]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:29.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:35:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:35:29.861 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:35:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:29] "GET /metrics HTTP/1.1" 200 191473 "" "Prometheus/2.33.4" 2024-03-12T03:35:30.067 INFO:teuthology.orchestra.run.smithi098.stdout:231928233999 2024-03-12T03:35:30.277 INFO:teuthology.orchestra.run.smithi098.stdout:55834574890 2024-03-12T03:35:30.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:30 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1246252093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:35:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:30 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1086595342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:35:30.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:30 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1246252093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:35:30.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:30 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1086595342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:35:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:30 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1246252093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-12T03:35:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:30 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1086595342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-12T03:35:31.018 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233996 got 231928233999 for osd.7 2024-03-12T03:35:31.019 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:31.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:31 smithi098 ceph-mon[24773]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:31.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:31 smithi098 ceph-mon[29473]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:31 smithi153 ceph-mon[25643]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:33.582 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:33.583 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:33.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:33 smithi098 ceph-mon[24773]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:33 smithi098 ceph-mon[29473]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:33 smithi153 ceph-mon[25643]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:33.963 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574887 got 55834574890 for osd.1 2024-03-12T03:35:33.963 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:34.951 INFO:teuthology.orchestra.run.smithi098.stdout:107374182433 2024-03-12T03:35:35.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:35 smithi098 ceph-mon[24773]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:35.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:35 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3288792289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:35:35.761 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:35 smithi098 ceph-mon[29473]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:35.761 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:35 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3288792289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:35:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:35 smithi153 ceph-mon[25643]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:35 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3288792289' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-12T03:35:36.362 INFO:teuthology.orchestra.run.smithi098.stdout:163208757273 2024-03-12T03:35:36.553 INFO:teuthology.orchestra.run.smithi098.stdout:128849018909 2024-03-12T03:35:36.669 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:36 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3089202194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:35:36.670 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:36 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3089202194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:35:36.892 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:36 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3089202194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-12T03:35:37.245 INFO:teuthology.orchestra.run.smithi098.stdout:197568495637 2024-03-12T03:35:37.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:37 smithi098 ceph-mon[24773]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:37.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:37 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2643653020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:35:37.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:37 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/39669744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:35:37.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:37 smithi098 ceph-mon[29473]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:37.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:37 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2643653020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:35:37.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:37 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/39669744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:35:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:37 smithi153 ceph-mon[25643]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:37 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2643653020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-12T03:35:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:37 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/39669744' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-12T03:35:38.787 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495637 for osd.6 2024-03-12T03:35:38.787 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:39.403 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182433 for osd.3 2024-03-12T03:35:39.403 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:39.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:39 smithi098 ceph-mon[24773]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:39 smithi098 ceph-mon[29473]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:39.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:39 smithi153 ceph-mon[25643]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:39.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:35:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:35:39.729 INFO:teuthology.orchestra.run.smithi098.stdout:77309411368 2024-03-12T03:35:39.983 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:35:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:39] "GET /metrics HTTP/1.1" 200 191473 "" "Prometheus/2.33.4" 2024-03-12T03:35:40.007 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018905 got 128849018909 for osd.4 2024-03-12T03:35:40.007 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:40.571 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:40 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3743843085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:35:40.571 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:40 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3743843085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:35:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:40 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3743843085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-12T03:35:41.114 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411365 got 77309411368 for osd.2 2024-03-12T03:35:41.115 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:41.677 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[24773]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[29473]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:35:41.678 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:35:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:41 smithi153 ceph-mon[25643]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:35:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:35:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:35:41.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:35:41.734 INFO:teuthology.orchestra.run.smithi098.stdout:34359738417 2024-03-12T03:35:42.002 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757270 got 163208757273 for osd.5 2024-03-12T03:35:42.002 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:42 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3032898170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:35:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:42 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3032898170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:35:42.688 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738417 for osd.0 2024-03-12T03:35:42.688 DEBUG:teuthology.parallel:result is None 2024-03-12T03:35:42.688 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-12T03:35:42.689 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:35:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:42 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3032898170' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-12T03:35:43.425 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:43.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:43 smithi098 ceph-mon[24773]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:43.675 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:43.675 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:43 smithi098 ceph-mon[29473]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:43.675 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:43 smithi153 ceph-mon[25643]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:45.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:44 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:44 smithi098 ceph-mon[29473]: Deploying daemon alertmanager.a on smithi098 2024-03-12T03:35:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:44 smithi098 ceph-mon[29473]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:44 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:44 smithi098 ceph-mon[24773]: Deploying daemon alertmanager.a on smithi098 2024-03-12T03:35:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:44 smithi098 ceph-mon[24773]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:45.163 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:35:45.164 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-12T03:35:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:44 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:44 smithi153 ceph-mon[25643]: Deploying daemon alertmanager.a on smithi098 2024-03-12T03:35:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:44 smithi153 ceph-mon[25643]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:45.853 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:45 smithi098 ceph-mon[29473]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:45.854 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:45 smithi098 ceph-mon[24773]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:45.875 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-03-12T03:35:44.466790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48908,"kb_used_data":6092,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912308,"statfs":{"total":767960285184,"available":767910203392,"internally_reserved":0,"allocated":6238208,"data_stored":3488659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002479"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.545247+0000","last_change":"2024-03-12T03:34:29.122477+0000","last_active":"2024-03-12T03:34:40.545247+0000","last_peered":"2024-03-12T03:34:40.545247+0000","last_clean":"2024-03-12T03:34:40.545247+0000","last_became_active":"2024-03-12T03:34:29.121949+0000","last_became_peered":"2024-03-12T03:34:29.121949+0000","last_unstale":"2024-03-12T03:34:40.545247+0000","last_undegraded":"2024-03-12T03:34:40.545247+0000","last_fullsized":"2024-03-12T03:34:40.545247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:32:32.649658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:32:32.649658+0000","last_clean_scrub_stamp":"2024-03-12T03:32:32.649658+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:19.845411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.627}]},{"osd":1,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.497},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.64300000000000002}]},{"osd":3,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77300000000000002}]},{"osd":4,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.81699999999999995}]},{"osd":5,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.873}]},{"osd":6,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.84699999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.503},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.58399999999999996}]},{"osd":1,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.65300000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.67000000000000004}]},{"osd":3,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.63200000000000001}]},{"osd":4,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.379},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.68200000000000005}]},{"osd":5,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.64300000000000002}]},{"osd":7,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.55600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.29599999999999999},"min":{"1min":0.25700000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.53200000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34000000000000002,"15min":0.317},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.45200000000000001}]},{"osd":2,"last update":"Tue Mar 12 03:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35899999999999999,"15min":0.34499999999999997},"min":{"1min":0.27300000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35099999999999998},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52700000000000002,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.39000000000000001}]},{"osd":3,"last update":"Tue Mar 12 03:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.42099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.50800000000000001,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.32800000000000001}]},{"osd":4,"last update":"Tue Mar 12 03:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.624}]},{"osd":5,"last update":"Tue Mar 12 03:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51500000000000001}]},{"osd":6,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Mar 12 03:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.498},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.66700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 12 03:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.33300000000000002,"15min":0.311},"min":{"1min":0.26800000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.32400000000000001},"min":{"1min":0.27300000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.60799999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.23200000000000001}]},{"osd":2,"last update":"Tue Mar 12 03:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36099999999999999},"min":{"1min":0.254,"5min":0.247,"15min":0.247},"max":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"last":0.251},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.38200000000000001,"15min":0.377},"min":{"1min":0.29399999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.318}]},{"osd":3,"last update":"Tue Mar 12 03:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42099999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.27300000000000002}]},{"osd":4,"last update":"Tue Mar 12 03:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75800000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.62}]},{"osd":5,"last update":"Tue Mar 12 03:34:38 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56799999999999995}]},{"osd":6,"last update":"Tue Mar 12 03:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.65600000000000003}]},{"osd":7,"last update":"Tue Mar 12 03:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.63600000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.46899999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.35799999999999998,"15min":0.34799999999999998},"min":{"1min":0.245,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.28499999999999998}]},{"osd":1,"last update":"Tue Mar 12 03:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.253},{"interface":"front","average":{"1min":0.371,"5min":0.376,"15min":0.36599999999999999},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.53900000000000003,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.30399999999999999}]},{"osd":3,"last update":"Tue Mar 12 03:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45700000000000002,"15min":0.46000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.437,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.32100000000000001}]},{"osd":4,"last update":"Tue Mar 12 03:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78500000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.47799999999999998}]},{"osd":5,"last update":"Tue Mar 12 03:34:40 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.499}]},{"osd":6,"last update":"Tue Mar 12 03:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54500000000000004}]},{"osd":7,"last update":"Tue Mar 12 03:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60699999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182435,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.376,"15min":0.375},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46500000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.26600000000000001}]},{"osd":1,"last update":"Tue Mar 12 03:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.371},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.316}]},{"osd":2,"last update":"Tue Mar 12 03:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41999999999999998,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.33200000000000002}]},{"osd":4,"last update":"Tue Mar 12 03:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.435},{"interface":"front","average":{"1min":0.624,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Tue Mar 12 03:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.42799999999999999}]},{"osd":6,"last update":"Tue Mar 12 03:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.38500000000000001}]},{"osd":7,"last update":"Tue Mar 12 03:35:31 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.46000000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.70499999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.139,"5min":0.69399999999999995,"15min":0.62},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":13.006,"5min":13.006,"15min":13.006},"last":0.51700000000000002}]},{"osd":1,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.70699999999999996,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":12.965999999999999,"5min":12.965999999999999,"15min":12.965999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.119,"5min":0.68700000000000006,"15min":0.61499999999999999},"min":{"1min":0.40600000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":12.945,"5min":12.945,"15min":12.945},"last":0.495}]},{"osd":2,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68100000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.69399999999999995,"5min":0.746,"15min":0.746},"last":0.55400000000000005}]},{"osd":3,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.41199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.65900000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73499999999999999,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.58299999999999996}]},{"osd":5,"last update":"Tue Mar 12 03:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.41499999999999998}]},{"osd":6,"last update":"Tue Mar 12 03:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Mar 12 03:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.434}]}]},{"osd":5,"up_from":38,"seq":163208757275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48099999999999998}]},{"osd":1,"last update":"Tue Mar 12 03:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.51100000000000001}]},{"osd":2,"last update":"Tue Mar 12 03:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55600000000000005}]},{"osd":3,"last update":"Tue Mar 12 03:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.53600000000000003}]},{"osd":4,"last update":"Tue Mar 12 03:34:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.435},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.45900000000000002}]},{"osd":6,"last update":"Tue Mar 12 03:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.51900000000000002}]},{"osd":7,"last update":"Tue Mar 12 03:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.501}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:35:45.877 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-12T03:35:46.086 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:45 smithi153 ceph-mon[25643]: from='client.24511 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:46 smithi098 ceph-mon[24773]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:46 smithi098 ceph-mon[29473]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:46 smithi153 ceph-mon[25643]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:49.021 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:35:49.023 INFO:teuthology.orchestra.run.smithi098.stderr:dumped all 2024-03-12T03:35:49.589 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:49 smithi098 ceph-mon[24773]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:49.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:49 smithi098 ceph-mon[29473]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:49 smithi153 ceph-mon[25643]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:49.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:35:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:35:50.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:35:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:49] "GET /metrics HTTP/1.1" 200 191469 "" "Prometheus/2.33.4" 2024-03-12T03:35:50.190 INFO:teuthology.orchestra.run.smithi098.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-12T03:35:48.467509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48908,"kb_used_data":6092,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912308,"statfs":{"total":767960285184,"available":767910203392,"internally_reserved":0,"allocated":6238208,"data_stored":3488659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002236"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-12T03:34:40.545247+0000","last_change":"2024-03-12T03:34:29.122477+0000","last_active":"2024-03-12T03:34:40.545247+0000","last_peered":"2024-03-12T03:34:40.545247+0000","last_clean":"2024-03-12T03:34:40.545247+0000","last_became_active":"2024-03-12T03:34:29.121949+0000","last_became_peered":"2024-03-12T03:34:29.121949+0000","last_unstale":"2024-03-12T03:34:40.545247+0000","last_undegraded":"2024-03-12T03:34:40.545247+0000","last_fullsized":"2024-03-12T03:34:40.545247+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-12T03:32:32.649658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-12T03:32:32.649658+0000","last_clean_scrub_stamp":"2024-03-12T03:32:32.649658+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-13T04:49:19.845411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6088,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739064,"statfs":{"total":95995035648,"available":95988801536,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.61599999999999999}]},{"osd":1,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.52400000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.60499999999999998}]},{"osd":3,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53500000000000003}]},{"osd":4,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65400000000000003}]},{"osd":5,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.629}]},{"osd":6,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.59699999999999998}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6148,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739004,"statfs":{"total":95995035648,"available":95988740096,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.50600000000000001}]},{"osd":1,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.442},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.52200000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.375}]},{"osd":3,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.39800000000000002}]},{"osd":4,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55600000000000005}]},{"osd":5,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.48899999999999999}]},{"osd":7,"last update":"Tue Mar 12 03:35:25 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.41799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6332,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738820,"statfs":{"total":95995035648,"available":95988551680,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:13 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.31900000000000001,"15min":0.29599999999999999},"min":{"1min":0.25700000000000001,"5min":0.185,"15min":0.185},"max":{"1min":0.53200000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.34000000000000002,"15min":0.317},"min":{"1min":0.26100000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.27200000000000002}]},{"osd":2,"last update":"Tue Mar 12 03:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35899999999999999,"15min":0.34499999999999997},"min":{"1min":0.27300000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.55500000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.314},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35099999999999998},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.52700000000000002,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.39400000000000002}]},{"osd":3,"last update":"Tue Mar 12 03:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.42099999999999999,"15min":0.42699999999999999},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.50800000000000001,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.378}]},{"osd":4,"last update":"Tue Mar 12 03:35:18 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57399999999999995,"15min":0.57199999999999995},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57799999999999996,"15min":0.58199999999999996},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.496}]},{"osd":5,"last update":"Tue Mar 12 03:35:40 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59899999999999998,"15min":0.60799999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.89100000000000001,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.75,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53700000000000003}]},{"osd":6,"last update":"Tue Mar 12 03:35:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.54700000000000004}]},{"osd":7,"last update":"Tue Mar 12 03:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.56100000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 12 03:35:14 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.33300000000000002,"15min":0.311},"min":{"1min":0.26800000000000002,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.34999999999999998,"15min":0.32400000000000001},"min":{"1min":0.27300000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.60799999999999998,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.47199999999999998}]},{"osd":2,"last update":"Tue Mar 12 03:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36699999999999999,"15min":0.36099999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.38,"15min":0.376},"min":{"1min":0.25600000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.52300000000000002,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.502}]},{"osd":3,"last update":"Tue Mar 12 03:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42199999999999999,"15min":0.42099999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.54700000000000004}]},{"osd":4,"last update":"Tue Mar 12 03:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.58099999999999996,"15min":0.58199999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.75800000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.58999999999999997,"15min":0.59199999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Mar 12 03:35:41 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.57299999999999995,"15min":0.57099999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.625},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59999999999999998,"15min":0.60099999999999998},"min":{"1min":0.45100000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.83899999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.57699999999999996}]},{"osd":6,"last update":"Tue Mar 12 03:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.85499999999999998}]},{"osd":7,"last update":"Tue Mar 12 03:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"last":0.83599999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35899999999999999,"15min":0.35699999999999998},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.46899999999999997,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.439},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.35799999999999998,"15min":0.34799999999999998},"min":{"1min":0.245,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.505}]},{"osd":1,"last update":"Tue Mar 12 03:35:37 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36799999999999999,"15min":0.36399999999999999},"min":{"1min":0.22,"5min":0.215,"15min":0.215},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.376,"15min":0.36599999999999999},"min":{"1min":0.25800000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.53900000000000003,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Tue Mar 12 03:34:59 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.45700000000000002,"15min":0.46000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.69599999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.437,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.45200000000000001}]},{"osd":4,"last update":"Tue Mar 12 03:35:24 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.56599999999999995,"15min":0.56899999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.78500000000000003,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.59699999999999998,"15min":0.59399999999999997},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56999999999999995}]},{"osd":5,"last update":"Tue Mar 12 03:35:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.80600000000000005,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.81200000000000006,"5min":0.874,"15min":0.874},"last":0.67400000000000004}]},{"osd":6,"last update":"Tue Mar 12 03:35:05 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.65700000000000003}]},{"osd":7,"last update":"Tue Mar 12 03:35:26 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.70399999999999996}]}]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.376,"15min":0.375},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46500000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57199999999999995,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.38400000000000001}]},{"osd":1,"last update":"Tue Mar 12 03:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.40500000000000003,"15min":0.40799999999999997},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.40400000000000003,"15min":0.40500000000000003},"min":{"1min":0.26400000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.51300000000000001}]},{"osd":2,"last update":"Tue Mar 12 03:35:00 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.59399999999999997,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.41999999999999998,"15min":0.42199999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.46400000000000002}]},{"osd":4,"last update":"Tue Mar 12 03:35:19 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.623},{"interface":"front","average":{"1min":0.624,"5min":0.56100000000000005,"15min":0.55100000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998}]},{"osd":5,"last update":"Tue Mar 12 03:34:43 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.747},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63}]},{"osd":6,"last update":"Tue Mar 12 03:35:07 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.66900000000000004}]},{"osd":7,"last update":"Tue Mar 12 03:35:31 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.78400000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018912,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.57399999999999995,"15min":0.58199999999999996},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.70499999999999996,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.139,"5min":0.69399999999999995,"15min":0.62},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":13.006,"5min":13.006,"15min":13.006},"last":0.61199999999999999}]},{"osd":1,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":0.70699999999999996,"15min":0.64100000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":12.965999999999999,"5min":12.965999999999999,"15min":12.965999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.119,"5min":0.68700000000000006,"15min":0.61499999999999999},"min":{"1min":0.40600000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":12.945,"5min":12.945,"15min":12.945},"last":0.496}]},{"osd":2,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.58199999999999996,"15min":0.59399999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.68100000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.56200000000000006,"15min":0.56799999999999995},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.69399999999999995,"5min":0.746,"15min":0.746},"last":0.53600000000000003}]},{"osd":3,"last update":"Tue Mar 12 03:35:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.41199999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.65900000000000003,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.58599999999999997,"15min":0.59599999999999997},"min":{"1min":0.436,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.73499999999999999,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.46899999999999997}]},{"osd":5,"last update":"Tue Mar 12 03:34:44 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64100000000000001}]},{"osd":6,"last update":"Tue Mar 12 03:35:04 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.59799999999999998}]},{"osd":7,"last update":"Tue Mar 12 03:35:28 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.622}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 12 03:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.52600000000000002,"15min":0.52400000000000002},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68200000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.495,"5min":0.51900000000000002,"15min":0.52300000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60799999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.50600000000000001}]},{"osd":1,"last update":"Tue Mar 12 03:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.53500000000000003,"15min":0.54000000000000004},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.73099999999999998,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.52400000000000002,"15min":0.52600000000000002},"min":{"1min":0.35999999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.70199999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Mar 12 03:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72999999999999998,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52000000000000002,"15min":0.51900000000000002},"min":{"1min":0.40899999999999997,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67900000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47999999999999998}]},{"osd":3,"last update":"Tue Mar 12 03:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.55400000000000005,"15min":0.55500000000000005},"min":{"1min":0.41599999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.53000000000000003,"15min":0.53100000000000003},"min":{"1min":0.34999999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71599999999999997,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.439}]},{"osd":4,"last update":"Tue Mar 12 03:35:46 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29199999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.34000000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69199999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Mar 12 03:35:02 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.53800000000000003}]},{"osd":7,"last update":"Tue Mar 12 03:35:30 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-12T03:35:50.191 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-12T03:35:50.192 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-12T03:35:50.192 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-12T03:35:50.192 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-12T03:35:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:50 smithi153 ceph-mon[25643]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:50.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:50 smithi098 ceph-mon[24773]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:50.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:50 smithi098 ceph-mon[29473]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-12T03:35:51.271 INFO:teuthology.orchestra.run.smithi098.stderr:Inferring config /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/mon.a/config 2024-03-12T03:35:51.670 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:51 smithi098 ceph-mon[24773]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:51.670 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:51 smithi098 ceph-mon[29473]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:51 smithi153 ceph-mon[25643]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:53.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:53 smithi098 ceph-mon[24773]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:53.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:53 smithi098 ceph-mon[29473]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:53 smithi153 ceph-mon[25643]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:54.405 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:35:54.405 INFO:teuthology.orchestra.run.smithi098.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-12T03:35:54.673 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:54 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/954758414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:35:54.673 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:54 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/954758414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:35:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:54 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/954758414' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-12T03:35:55.638 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-12T03:35:55.639 INFO:tasks.cephadm:Setup complete, yielding 2024-03-12T03:35:55.639 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:35:55.650 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-12T03:35:55.650 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-03-12T03:35:55.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[24773]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[29473]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:35:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:55 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:55 smithi153 ceph-mon[25643]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:55 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:55 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:55 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:55 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:35:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:55 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:56 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:35:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:56 smithi098 ceph-mon[29473]: Deploying daemon grafana.a on smithi153 2024-03-12T03:35:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:56 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:56 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:35:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:56 smithi098 ceph-mon[24773]: Deploying daemon grafana.a on smithi153 2024-03-12T03:35:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:56 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:56.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:35:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[47968]: level=info ts=2024-03-12T03:35:56.776Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000439216s 2024-03-12T03:35:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:56 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:35:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:56 smithi153 ceph-mon[25643]: Deploying daemon grafana.a on smithi153 2024-03-12T03:35:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:56 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:35:57.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:57 smithi098 ceph-mon[24773]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:57.773 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:57 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' 2024-03-12T03:35:57.774 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:57 smithi098 ceph-mon[29473]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:57.774 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:57 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' 2024-03-12T03:35:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:57 smithi153 ceph-mon[25643]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:57 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' 2024-03-12T03:35:58.056 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T03:35:58.065 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-12T03:35:58.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-12T03:35:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:35:59 smithi098 ceph-mon[24773]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:59.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:35:59 smithi098 ceph-mon[29473]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:59.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:35:59 smithi153 ceph-mon[25643]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:35:59.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:35:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:36:00.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:35:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:35:59] "GET /metrics HTTP/1.1" 200 191481 "" "Prometheus/2.33.4" 2024-03-12T03:36:01.458 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:36:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "id": "58a2f794-e8dc-4883-aadf-2eba8246c82e", 2024-03-12T03:36:01.458 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "r", 2024-03-12T03:36:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "current_period": "31ba3d23-4dc1-43cf-ac0d-0ff2dee73406", 2024-03-12T03:36:01.459 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1 2024-03-12T03:36:01.459 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:36:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:01 smithi153 ceph-mon[25643]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:36:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:01 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1111409858' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:36:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:01 smithi153 ceph-mon[25643]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:01.740 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:01 smithi098 ceph-mon[24773]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:36:01.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:01 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1111409858' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:36:01.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:01 smithi098 ceph-mon[24773]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:01.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:01 smithi098 ceph-mon[29473]: osdmap e60: 8 total, 8 up, 8 in 2024-03-12T03:36:01.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:01 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1111409858' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-12T03:36:01.741 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:01 smithi098 ceph-mon[29473]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:02.056 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-12T03:36:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:02 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1111409858' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:36:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:02 smithi153 ceph-mon[25643]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:36:02.693 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:02 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1111409858' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:36:02.693 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:02 smithi098 ceph-mon[24773]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:36:02.693 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:02 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1111409858' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-12T03:36:02.693 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:02 smithi098 ceph-mon[29473]: osdmap e61: 8 total, 8 up, 8 in 2024-03-12T03:36:03.635 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:03 smithi098 ceph-mon[24773]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:36:03.636 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:03 smithi098 ceph-mon[24773]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:03.636 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:03 smithi098 ceph-mon[29473]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:36:03.636 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:03 smithi098 ceph-mon[29473]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:03.668 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "id": "a06f9e3c-0923-4c68-89ca-aaa937d5bdd0", 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default", 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "api_name": "default", 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "is_master": "true", 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "endpoints": [], 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "hostnames": [], 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "hostnames_s3website": [], 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "master_zone": "", 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "zones": [], 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "placement_targets": [], 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "default_placement": "", 2024-03-12T03:36:03.669 INFO:teuthology.orchestra.run.smithi098.stdout: "realm_id": "58a2f794-e8dc-4883-aadf-2eba8246c82e", 2024-03-12T03:36:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_policy": { 2024-03-12T03:36:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: "groups": [] 2024-03-12T03:36:03.670 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:03.670 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:36:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:03 smithi153 ceph-mon[25643]: osdmap e62: 8 total, 8 up, 8 in 2024-03-12T03:36:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:03 smithi153 ceph-mon[25643]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:04.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-12T03:36:05.034 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[47968]: level=info ts=2024-03-12T03:36:04.778Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001716683s 2024-03-12T03:36:05.788 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:05 smithi098 ceph-mon[24773]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:05.788 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:05 smithi098 ceph-mon[29473]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:05 smithi153 ceph-mon[25643]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "id": "ee756faf-e01c-4e38-ad50-e963f846023a", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "domain_root": "z.rgw.meta:root", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "control_pool": "z.rgw.control", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "log_pool": "z.rgw.log", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-12T03:36:06.045 INFO:teuthology.orchestra.run.smithi098.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "otp_pool": "z.rgw.otp", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "system_key": { 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "access_key": "", 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: "secret_key": "" 2024-03-12T03:36:06.046 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "placement_pools": [ 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "default-placement", 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "val": { 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_classes": { 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "STANDARD": { 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: "index_type": 0 2024-03-12T03:36:06.047 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:36:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: "realm_id": "58a2f794-e8dc-4883-aadf-2eba8246c82e", 2024-03-12T03:36:06.048 INFO:teuthology.orchestra.run.smithi098.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-12T03:36:06.048 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:36:06.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-12T03:36:07.785 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:07 smithi098 ceph-mon[24773]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:07.785 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:07 smithi098 ceph-mon[29473]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:07.865 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:07 smithi153 ceph-mon[25643]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-12T03:36:09.643 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:36:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:36:09.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:09 smithi153 ceph-mon[25643]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-12T03:36:09.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:09 smithi153 ceph-mon[25643]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:36:09.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:09 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3177763135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:36:09.644 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:09 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:36:09.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:09] "GET /metrics HTTP/1.1" 200 191481 "" "Prometheus/2.33.4" 2024-03-12T03:36:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[24773]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-12T03:36:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[24773]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:36:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3177763135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:36:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:36:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[29473]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-12T03:36:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[29473]: osdmap e63: 8 total, 8 up, 8 in 2024-03-12T03:36:09.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3177763135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:36:09.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:09 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-12T03:36:10.556 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-12T03:36:10.556 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-mon[25643]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:36:10.556 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-12T03:36:10.557 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-12T03:36:10.811 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-12T03:36:10.812 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-12T03:36:10.813 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-12T03:36:10.814 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-12T03:36:10.815 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-12T03:36:10.816 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-12T03:36:10.816 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-12T03:36:10.816 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-12T03:36:10.816 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-12T03:36:10.817 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-12T03:36:10.818 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-12T03:36:10.819 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-12T03:36:10.820 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-12T03:36:10.821 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-12T03:36:10.822 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-12T03:36:10.823 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-12T03:36:10.824 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-12T03:36:10.825 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-12T03:36:10.826 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-12T03:36:10.827 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:10 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-12T03:36:10.827 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:10 smithi098 ceph-mon[24773]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:36:10.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:10 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-12T03:36:10.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:10 smithi098 ceph-mon[29473]: osdmap e64: 8 total, 8 up, 8 in 2024-03-12T03:36:11.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-12T03:36:11.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-12T03:36:11.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-12T03:36:11.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-12T03:36:11.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-12T03:36:11.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-12T03:36:11.095 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-12T03:36:11.096 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-12T03:36:11.097 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-12T03:36:11.098 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-12T03:36:11.099 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-12T03:36:11.100 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-12T03:36:11.101 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-12T03:36:11.102 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-12T03:36:11.103 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=547.582853ms 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-12T03:36:11.104 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-12T03:36:11.105 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-12T03:36:11.105 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-12T03:36:11.105 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-12T03:36:11.105 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-12T03:36:11.105 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-12T03:36:11.105 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-12T03:36:11.435 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-12T03:36:11.435 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:36:11+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-12T03:36:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:36:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:36:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3177763135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:36:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:36:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:11 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:11.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:36:11.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:11.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:36:11.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3177763135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:36:11.808 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 1023 B/s wr, 2 op/s 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: osdmap e65: 8 total, 8 up, 8 in 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3177763135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:11.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:11 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:12.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:12 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-12T03:36:12.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:12 smithi098 ceph-mon[24773]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:36:12.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:12 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-12T03:36:12.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:12 smithi098 ceph-mon[29473]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:36:12.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:12 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-12T03:36:12.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:12 smithi153 ceph-mon[25643]: osdmap e66: 8 total, 8 up, 8 in 2024-03-12T03:36:13.857 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:13 smithi153 ceph-mon[25643]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-12T03:36:13.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:13 smithi153 ceph-mon[25643]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:36:13.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:13 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:36:13.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:13 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:36:13.858 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:13 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[24773]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[24773]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[29473]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[29473]: osdmap e67: 8 total, 8 up, 8 in 2024-03-12T03:36:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:36:14.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-12T03:36:14.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:13 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:14.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:36:14.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[29473]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:36:14.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:36:14.933 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:36:14.934 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:14.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:36:14.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[24773]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:36:14.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:36:14.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:36:14.934 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:14 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:14 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-12T03:36:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:14 smithi153 ceph-mon[25643]: osdmap e68: 8 total, 8 up, 8 in 2024-03-12T03:36:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:14 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:36:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:14 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-12T03:36:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:14 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:15.636 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:36:15.636 INFO:teuthology.orchestra.run.smithi098.stdout: "id": "7b26b0fe-b147-4cc5-8cb3-521783d8fe67", 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "epoch": 1, 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "predecessor_uuid": "31ba3d23-4dc1-43cf-ac0d-0ff2dee73406", 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_status": [], 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "period_map": { 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "id": "7b26b0fe-b147-4cc5-8cb3-521783d8fe67", 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "zonegroups": [ 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "id": "a06f9e3c-0923-4c68-89ca-aaa937d5bdd0", 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default", 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "api_name": "default", 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "is_master": "true", 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "endpoints": [], 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "hostnames": [], 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "hostnames_s3website": [], 2024-03-12T03:36:15.637 INFO:teuthology.orchestra.run.smithi098.stdout: "master_zone": "ee756faf-e01c-4e38-ad50-e963f846023a", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "zones": [ 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "id": "ee756faf-e01c-4e38-ad50-e963f846023a", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "z", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "endpoints": [], 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "log_meta": "false", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "log_data": "false", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "bucket_index_max_shards": 11, 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "read_only": "false", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "tier_type": "", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_from_all": "true", 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_from": [], 2024-03-12T03:36:15.638 INFO:teuthology.orchestra.run.smithi098.stdout: "redirect_zone": "" 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "placement_targets": [ 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "name": "default-placement", 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "tags": [], 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "storage_classes": [ 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "STANDARD" 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "default_placement": "default-placement", 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "realm_id": "58a2f794-e8dc-4883-aadf-2eba8246c82e", 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "sync_policy": { 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: "groups": [] 2024-03-12T03:36:15.639 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "short_zone_ids": [ 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: { 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "key": "ee756faf-e01c-4e38-ad50-e963f846023a", 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "val": 720321927 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: ] 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "master_zonegroup": "a06f9e3c-0923-4c68-89ca-aaa937d5bdd0", 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "master_zone": "ee756faf-e01c-4e38-ad50-e963f846023a", 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "period_config": { 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "bucket_quota": { 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": false, 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "check_on_raw": false, 2024-03-12T03:36:15.640 INFO:teuthology.orchestra.run.smithi098.stdout: "max_size": -1, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_size_kb": 0, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_objects": -1 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "user_quota": { 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": false, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "check_on_raw": false, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_size": -1, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_size_kb": 0, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_objects": -1 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "user_ratelimit": { 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_read_ops": 0, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_write_ops": 0, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_read_bytes": 0, 2024-03-12T03:36:15.641 INFO:teuthology.orchestra.run.smithi098.stdout: "max_write_bytes": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": false 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "bucket_ratelimit": { 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_read_ops": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_write_ops": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_read_bytes": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_write_bytes": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": false 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "anonymous_ratelimit": { 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_read_ops": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_write_ops": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_read_bytes": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "max_write_bytes": 0, 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: "enabled": false 2024-03-12T03:36:15.642 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:36:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:36:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "realm_id": "58a2f794-e8dc-4883-aadf-2eba8246c82e", 2024-03-12T03:36:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "realm_name": "r", 2024-03-12T03:36:15.643 INFO:teuthology.orchestra.run.smithi098.stdout: "realm_epoch": 2 2024-03-12T03:36:15.643 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:36:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-12T03:36:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:36:15.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:36:15.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:15.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-12T03:36:15.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-12T03:36:15.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:36:15.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: Reconfiguring daemon alertmanager.a on smithi098 2024-03-12T03:36:15.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:15 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:16.028 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-12T03:36:16.028 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:36:16.028 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: Reconfiguring daemon alertmanager.a on smithi098 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: osdmap e69: 8 total, 8 up, 8 in 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/4240100110' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-12T03:36:16.029 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-12T03:36:16.030 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:36:16.030 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: Reconfiguring daemon alertmanager.a on smithi098 2024-03-12T03:36:16.030 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:15 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:16.400 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-12T03:36:16.770 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:16 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-12T03:36:16.770 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:16 smithi098 ceph-mon[29473]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:36:16.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:16 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-12T03:36:16.771 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:16 smithi098 ceph-mon[24773]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:36:16.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:16 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-12T03:36:16.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:16 smithi153 ceph-mon[25643]: osdmap e70: 8 total, 8 up, 8 in 2024-03-12T03:36:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:17 smithi098 ceph-mon[29473]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:17 smithi098 ceph-mon[24773]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:17 smithi153 ceph-mon[25643]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:18.252 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:18 smithi098 systemd[1]: Stopping Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:36:18.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:18 smithi098 bash[49392]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-12T03:36:19.058 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[47968]: level=info ts=2024-03-12T03:36:18.793Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:36:19.323 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:19 smithi098 podman[49408]: 2024-03-12 03:36:19.253910254 +0000 UTC m=+0.994269161 container died b64440a6d0a634cc5ba42d709bce6c588351ce106108a0de8b322bc85cd301e1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:19 smithi153 ceph-mon[25643]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:36:19.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:36:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:36:19.715 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:19 smithi098 ceph-mon[24773]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:36:19.715 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:19 smithi098 ceph-mon[29473]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-12T03:36:19.715 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:36:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:19] "GET /metrics HTTP/1.1" 200 194250 "" "Prometheus/2.33.4" 2024-03-12T03:36:19.716 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:19 smithi098 podman[49408]: 2024-03-12 03:36:19.58753526 +0000 UTC m=+1.327894183 container cleanup b64440a6d0a634cc5ba42d709bce6c588351ce106108a0de8b322bc85cd301e1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:19.716 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:19 smithi098 bash[49408]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:36:20.000 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:19 smithi098 bash[49470]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-12T03:36:20.220 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled rgw.foo update... 2024-03-12T03:36:20.764 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:20 smithi098 podman[49440]: 2024-03-12 03:36:20.38412573 +0000 UTC m=+1.555010793 container remove b64440a6d0a634cc5ba42d709bce6c588351ce106108a0de8b322bc85cd301e1 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:20.764 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:20 smithi098 podman[49440]: 2024-03-12 03:36:20.442869787 +0000 UTC m=+1.613754804 volume remove fb5afc1a5c85d850f3efb49d6c2b28f1e403a1f0d30e3d3f016b8dfdcad7fa8d 2024-03-12T03:36:20.764 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:20 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:36:20.764 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:20 smithi098 systemd[1]: Stopped Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:36:20.764 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:20 smithi098 systemd[1]: Starting Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:36:21.064 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph osd pool create foo' 2024-03-12T03:36:21.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[29473]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:36:21.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[29473]: Saving service rgw.foo spec with placement count:2 2024-03-12T03:36:21.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:21.291 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[29473]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:36:21.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[24773]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:36:21.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[24773]: Saving service rgw.foo spec with placement count:2 2024-03-12T03:36:21.291 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:21.292 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-mon[24773]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:36:21.573 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 podman[49596]: 2024-03-12 03:36:21.289137592 +0000 UTC m=+0.118434772 volume create cddf350d208d22949551d0f9fa665be999d7e04ad99aa40bf98eaa8cf1cb49d1 2024-03-12T03:36:21.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 podman[49596]: 2024-03-12 03:36:21.194003819 +0000 UTC m=+0.023301002 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-12T03:36:21.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 podman[49596]: 2024-03-12 03:36:21.406267386 +0000 UTC m=+0.235564565 container create c12f6c606e5fc4254907ddc9fe3a4ca0a32c2ed9a2b776635ad8075ff2d3aa0c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:21 smithi153 ceph-mon[25643]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:36:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:21 smithi153 ceph-mon[25643]: Saving service rgw.foo spec with placement count:2 2024-03-12T03:36:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:21 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:21 smithi153 ceph-mon[25643]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 podman[49596]: 2024-03-12 03:36:21.767833249 +0000 UTC m=+0.597130441 container init c12f6c606e5fc4254907ddc9fe3a4ca0a32c2ed9a2b776635ad8075ff2d3aa0c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 podman[49596]: 2024-03-12 03:36:21.770388254 +0000 UTC m=+0.599685445 container start c12f6c606e5fc4254907ddc9fe3a4ca0a32c2ed9a2b776635ad8075ff2d3aa0c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.780Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.780Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.782Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.98 port=9094 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.783Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.807Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:36:21.864 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.807Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:36:21.865 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.809Z caller=main.go:518 msg=Listening address=:9093 2024-03-12T03:36:21.865 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:21.809Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-12T03:36:21.865 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 bash[49596]: c12f6c606e5fc4254907ddc9fe3a4ca0a32c2ed9a2b776635ad8075ff2d3aa0c 2024-03-12T03:36:22.155 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:21 smithi098 systemd[1]: Started Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:36:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[24773]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:36:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[24773]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:36:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[24773]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-12T03:36:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[29473]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:36:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[29473]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:36:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:22 smithi098 ceph-mon[29473]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-12T03:36:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:22 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:22 smithi153 ceph-mon[25643]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:36:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:22 smithi153 ceph-mon[25643]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:36:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:22 smithi153 ceph-mon[25643]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-12T03:36:24.073 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:23.783Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000097567s 2024-03-12T03:36:24.437 INFO:teuthology.orchestra.run.smithi098.stderr:pool 'foo' created 2024-03-12T03:36:24.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:24 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/395416084' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-12T03:36:24.757 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:24 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/395416084' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-12T03:36:24.758 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:24 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/395416084' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-12T03:36:25.150 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'rbd pool init foo' 2024-03-12T03:36:25.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:25 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/395416084' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-12T03:36:25.687 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:25 smithi098 ceph-mon[24773]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:36:25.688 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:25 smithi098 ceph-mon[24773]: pgmap v67: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-12T03:36:25.688 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:25 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/395416084' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-12T03:36:25.688 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:25 smithi098 ceph-mon[29473]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:36:25.688 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:25 smithi098 ceph-mon[29473]: pgmap v67: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-12T03:36:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:25 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/395416084' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-12T03:36:25.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:25 smithi153 ceph-mon[25643]: osdmap e71: 8 total, 8 up, 8 in 2024-03-12T03:36:25.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:25 smithi153 ceph-mon[25643]: pgmap v67: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 1.0 KiB/s wr, 2 op/s 2024-03-12T03:36:26.431 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 systemd[1]: Stopping Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 bash[42440]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.283Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.283Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.283Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.283Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.283Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.283Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.284Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.284Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-12T03:36:26.432 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.284Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:36:26.433 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.284Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:36:26.433 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.284Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-12T03:36:26.433 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[40776]: ts=2024-03-12T03:36:26.284Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-12T03:36:26.433 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 podman[42453]: 2024-03-12 03:36:26.2940997 +0000 UTC m=+0.109771992 container died 6086ef920aa3383c3d375b4b2b5b64198cd759e26e5ecd21dc3d518263311639 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:26.686 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-mon[25643]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:36:26.686 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:26 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:26.686 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 podman[42453]: 2024-03-12 03:36:26.650034943 +0000 UTC m=+0.465707218 container cleanup 6086ef920aa3383c3d375b4b2b5b64198cd759e26e5ecd21dc3d518263311639 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:26.686 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 bash[42453]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:36:26.687 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 bash[42480]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:36:26.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:26 smithi098 ceph-mon[24773]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:36:26.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:26 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:26.747 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:26 smithi098 ceph-mon[29473]: osdmap e72: 8 total, 8 up, 8 in 2024-03-12T03:36:26.747 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:26 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:27.275 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 podman[42467]: 2024-03-12 03:36:26.95090101 +0000 UTC m=+0.658955884 container remove 6086ef920aa3383c3d375b4b2b5b64198cd759e26e5ecd21dc3d518263311639 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:27.275 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:36:27.275 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 systemd[1]: Stopped Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:36:27.275 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:26 smithi153 systemd[1]: Starting Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:36:27.275 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 podman[42553]: 2024-03-12 03:36:27.176852994 +0000 UTC m=+0.107236842 container create 7edcfd1725dd9b292c4bbb9c2411ba7ff3a9ed29d19aa5ee0c7118fd3afbb9e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:27.275 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 podman[42553]: 2024-03-12 03:36:27.088413611 +0000 UTC m=+0.018797468 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-12T03:36:27.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-mon[25643]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:36:27.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-mon[25643]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:27.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/710643445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-12T03:36:27.623 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 podman[42553]: 2024-03-12 03:36:27.362371832 +0000 UTC m=+0.292755673 container init 7edcfd1725dd9b292c4bbb9c2411ba7ff3a9ed29d19aa5ee0c7118fd3afbb9e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 podman[42553]: 2024-03-12 03:36:27.364925603 +0000 UTC m=+0.295309443 container start 7edcfd1725dd9b292c4bbb9c2411ba7ff3a9ed29d19aa5ee0c7118fd3afbb9e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.388Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.388Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.388Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.388Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi153 (none))" 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.388Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.388Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:36:27.624 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.389Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.390Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.391Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.393Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.393Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.716µs 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.393Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 bash[42553]: 7edcfd1725dd9b292c4bbb9c2411ba7ff3a9ed29d19aa5ee0c7118fd3afbb9e7 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 systemd[1]: Started Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.599Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.600Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.600Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.177µs wal_replay_duration=206.75977ms total_replay_duration=206.799261ms 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.602Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.602Z caller=main.go:947 level=info msg="TSDB started" 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.602Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.621Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.231606ms db_storage=758ns remote_storage=1.58µs web_handler=368ns query_engine=636ns scrape=377.596µs scrape_sd=26.434µs notify=25.199µs notify_sd=8.293µs rules=18.499411ms 2024-03-12T03:36:27.625 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:36:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:36:27.621Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[24773]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[24773]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/710643445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[29473]: osdmap e73: 8 total, 8 up, 8 in 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[29473]: pgmap v70: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/710643445' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-12T03:36:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:27 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.98:9093"}]: dispatch 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.98:9093"}]: dispatch 2024-03-12T03:36:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.98:9093"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.98:9093"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:36:28.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:28.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:28 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:28.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-12T03:36:28.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: osdmap e74: 8 total, 8 up, 8 in 2024-03-12T03:36:28.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:36:28.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:36:28.919 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.98:9093"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.98:9093"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.153:9095"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.153:3000"}]: dispatch 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:28.920 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:28.921 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:28 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:29.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:29 smithi153 ceph-mon[25643]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:36:29.731 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:29 smithi153 ceph-mon[25643]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:29.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:29 smithi098 ceph-mon[24773]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:36:29.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:29 smithi098 ceph-mon[24773]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:29 smithi098 ceph-mon[29473]: osdmap e75: 8 total, 8 up, 8 in 2024-03-12T03:36:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:29 smithi098 ceph-mon[29473]: pgmap v73: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:30.616 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-12T03:36:30.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:30 smithi153 ceph-mon[25643]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:36:30.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:30 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:30.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:30 smithi098 ceph-mon[24773]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:36:30.770 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:30 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:30.770 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:30 smithi098 ceph-mon[29473]: osdmap e76: 8 total, 8 up, 8 in 2024-03-12T03:36:30.770 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:30 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.548 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.549 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:31.823 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:36:31 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:36:31.786Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002985723s 2024-03-12T03:36:31.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:31.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:31.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:31.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:36:31.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:31.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:31 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:32.565 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled iscsi.foo update... 2024-03-12T03:36:32.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:32 smithi098 ceph-mon[29473]: Saving service rgw.foo spec with placement count:2 2024-03-12T03:36:32.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:32 smithi098 ceph-mon[29473]: Deploying daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T03:36:32.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:32 smithi098 ceph-mon[24773]: Saving service rgw.foo spec with placement count:2 2024-03-12T03:36:32.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:32 smithi098 ceph-mon[24773]: Deploying daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T03:36:32.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:32 smithi153 ceph-mon[25643]: Saving service rgw.foo spec with placement count:2 2024-03-12T03:36:32.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:32 smithi153 ceph-mon[25643]: Deploying daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T03:36:33.460 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'sleep 180' 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[29473]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[29473]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[29473]: Saving service iscsi.foo spec with placement count:1 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[24773]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[24773]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[24773]: Saving service iscsi.foo spec with placement count:1 2024-03-12T03:36:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:33 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:33 smithi153 ceph-mon[25643]: pgmap v76: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 341 B/s wr, 0 op/s 2024-03-12T03:36:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:33 smithi153 ceph-mon[25643]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:36:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:33 smithi153 ceph-mon[25643]: Saving service iscsi.foo spec with placement count:1 2024-03-12T03:36:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:33 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:35.708 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:35 smithi098 ceph-mon[24773]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 291 B/s wr, 0 op/s 2024-03-12T03:36:35.708 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:35 smithi098 ceph-mon[29473]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 291 B/s wr, 0 op/s 2024-03-12T03:36:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:35 smithi153 ceph-mon[25643]: pgmap v77: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 291 B/s wr, 0 op/s 2024-03-12T03:36:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: Deploying daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[24773]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 254 B/s wr, 0 op/s 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:36:37.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:37.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:37.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: Deploying daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T03:36:37.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:37 smithi098 ceph-mon[29473]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 254 B/s wr, 0 op/s 2024-03-12T03:36:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T03:36:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-12T03:36:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: Deploying daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T03:36:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:37 smithi153 ceph-mon[25643]: pgmap v78: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 254 B/s wr, 0 op/s 2024-03-12T03:36:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[24773]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:36:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[24773]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 3.9 KiB/s wr, 153 op/s 2024-03-12T03:36:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[29473]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:36:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[29473]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 3.9 KiB/s wr, 153 op/s 2024-03-12T03:36:39.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:39.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:39.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:39.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:36:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:36:39.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:39 smithi153 ceph-mon[25643]: osdmap e77: 8 total, 8 up, 8 in 2024-03-12T03:36:39.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:39 smithi153 ceph-mon[25643]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s rd, 3.9 KiB/s wr, 153 op/s 2024-03-12T03:36:39.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:39 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:39.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:39 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:39.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:39 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:40.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:36:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:39] "GET /metrics HTTP/1.1" 200 197894 "" "Prometheus/2.33.4" 2024-03-12T03:36:40.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-03-12T03:36:40.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-03-12T03:36:40.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:36:40.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.812 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:36:40.813 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.814 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-12T03:36:40.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-03-12T03:36:40.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:36:40.815 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 3.5 KiB/s wr, 138 op/s 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:36:41.775 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:41.776 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:41.776 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:41 smithi153 ceph-mon[25643]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 3.5 KiB/s wr, 138 op/s 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:41.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[24773]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 62 KiB/s rd, 3.5 KiB/s wr, 138 op/s 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: osdmap e78: 8 total, 8 up, 8 in 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:41.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:41 smithi098 ceph-mon[29473]: osdmap e79: 8 total, 8 up, 8 in 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: Checking dashboard <-> RGW credentials 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: Checking pool "foo" exists for service iscsi.foo 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: Deploying daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:36:43.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:43 smithi153 ceph-mon[25643]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 292 KiB/s rd, 5.5 KiB/s wr, 522 op/s 2024-03-12T03:36:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: Checking dashboard <-> RGW credentials 2024-03-12T03:36:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: Checking pool "foo" exists for service iscsi.foo 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: Deploying daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[24773]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 292 KiB/s rd, 5.5 KiB/s wr, 522 op/s 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: Checking dashboard <-> RGW credentials 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:43.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: Checking pool "foo" exists for service iscsi.foo 2024-03-12T03:36:43.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:36:43.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:36:43.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-12T03:36:43.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:43.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: Deploying daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:36:43.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:43 smithi098 ceph-mon[29473]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 292 KiB/s rd, 5.5 KiB/s wr, 522 op/s 2024-03-12T03:36:44.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:44 smithi098 ceph-mon[24773]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-12T03:36:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:44 smithi098 ceph-mon[29473]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-12T03:36:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:44 smithi153 ceph-mon[25643]: mgrmap e20: y(active, since 2m), standbys: x 2024-03-12T03:36:45.519 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[24773]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 0 B/s wr, 289 op/s 2024-03-12T03:36:45.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:45.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:45.520 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:45.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[29473]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 0 B/s wr, 289 op/s 2024-03-12T03:36:45.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:45.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:45.520 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:45 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:45 smithi153 ceph-mon[25643]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 187 KiB/s rd, 0 B/s wr, 289 op/s 2024-03-12T03:36:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:45 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:45 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:45 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:46.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:46 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:46.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:46 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2720574801' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:36:46.782 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:46 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3371302418' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3339782884"}]: dispatch 2024-03-12T03:36:46.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:46 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:46.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:46 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2720574801' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:36:46.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:46 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3371302418' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3339782884"}]: dispatch 2024-03-12T03:36:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:46 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:46 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2720574801' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:36:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:46 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3371302418' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3339782884"}]: dispatch 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[24773]: pgmap v86: 161 pgs: 3 peering, 158 active+clean; 584 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 127 B/s wr, 225 op/s 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3371302418' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3339782884"}]': finished 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[24773]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3938747528' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]: dispatch 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]: dispatch 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[29473]: pgmap v86: 161 pgs: 3 peering, 158 active+clean; 584 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 127 B/s wr, 225 op/s 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3371302418' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3339782884"}]': finished 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[29473]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:36:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3938747528' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]: dispatch 2024-03-12T03:36:47.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:47 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]: dispatch 2024-03-12T03:36:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:47 smithi153 ceph-mon[25643]: pgmap v86: 161 pgs: 3 peering, 158 active+clean; 584 KiB data, 66 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 127 B/s wr, 225 op/s 2024-03-12T03:36:47.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:47 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3371302418' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3339782884"}]': finished 2024-03-12T03:36:47.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:47 smithi153 ceph-mon[25643]: osdmap e80: 8 total, 8 up, 8 in 2024-03-12T03:36:47.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:47 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3938747528' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]: dispatch 2024-03-12T03:36:47.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:47 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]: dispatch 2024-03-12T03:36:48.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:48 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]': finished 2024-03-12T03:36:48.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:48 smithi153 ceph-mon[25643]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:36:48.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:48 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2336667071' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]: dispatch 2024-03-12T03:36:48.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:48 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]: dispatch 2024-03-12T03:36:49.024 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]': finished 2024-03-12T03:36:49.024 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[29473]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:36:49.024 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2336667071' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]: dispatch 2024-03-12T03:36:49.024 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]: dispatch 2024-03-12T03:36:49.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1032522663"}]': finished 2024-03-12T03:36:49.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[24773]: osdmap e81: 8 total, 8 up, 8 in 2024-03-12T03:36:49.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2336667071' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]: dispatch 2024-03-12T03:36:49.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:48 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]: dispatch 2024-03-12T03:36:49.641 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:49] "GET /metrics HTTP/1.1" 200 206464 "" "Prometheus/2.33.4" 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 590 B/s wr, 27 op/s; 51 B/s, 0 objects/s recovering 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]': finished 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2339168725' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3469331030"}]: dispatch 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:36:49.642 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:36:49.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi098"}]: dispatch 2024-03-12T03:36:49.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:49.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:49.643 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.643 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 590 B/s wr, 27 op/s; 51 B/s, 0 objects/s recovering 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]': finished 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2339168725' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3469331030"}]: dispatch 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi098"}]: dispatch 2024-03-12T03:36:49.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:49.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:49.645 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:49 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:36:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 590 B/s wr, 27 op/s; 51 B/s, 0 objects/s recovering 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3485535046"}]': finished 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: osdmap e82: 8 total, 8 up, 8 in 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2339168725' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3469331030"}]: dispatch 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi098"}]: dispatch 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:36:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:49 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:36:50.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: Adding iSCSI gateway http://:@172.21.15.98:5000 to Dashboard 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi098"}]: dispatch 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: Checking dashboard <-> RGW credentials 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2339168725' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3469331030"}]': finished 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/185864377' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]: dispatch 2024-03-12T03:36:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:50 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]: dispatch 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: Adding iSCSI gateway http://:@172.21.15.98:5000 to Dashboard 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi098"}]: dispatch 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: Checking dashboard <-> RGW credentials 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2339168725' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3469331030"}]': finished 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/185864377' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]: dispatch 2024-03-12T03:36:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]: dispatch 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: Adding iSCSI gateway http://:@172.21.15.98:5000 to Dashboard 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi098"}]: dispatch 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: Checking dashboard <-> RGW credentials 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2339168725' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3469331030"}]': finished 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: osdmap e83: 8 total, 8 up, 8 in 2024-03-12T03:36:51.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/185864377' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]: dispatch 2024-03-12T03:36:51.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:50 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]: dispatch 2024-03-12T03:36:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:51 smithi153 ceph-mon[25643]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 767 B/s wr, 34 op/s; 89 B/s, 0 objects/s recovering 2024-03-12T03:36:51.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:51 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]': finished 2024-03-12T03:36:51.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:51 smithi153 ceph-mon[25643]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:36:51.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:51 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1303016910' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/85873921"}]: dispatch 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[24773]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 767 B/s wr, 34 op/s; 89 B/s, 0 objects/s recovering 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]': finished 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[24773]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1303016910' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/85873921"}]: dispatch 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[29473]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 74 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 767 B/s wr, 34 op/s; 89 B/s, 0 objects/s recovering 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1041454473"}]': finished 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[29473]: osdmap e84: 8 total, 8 up, 8 in 2024-03-12T03:36:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:51 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1303016910' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/85873921"}]: dispatch 2024-03-12T03:36:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:52 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1303016910' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/85873921"}]': finished 2024-03-12T03:36:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:52 smithi153 ceph-mon[25643]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:36:52.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:52 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3106503942' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1041454473"}]: dispatch 2024-03-12T03:36:53.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:52 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1303016910' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/85873921"}]': finished 2024-03-12T03:36:53.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:52 smithi098 ceph-mon[24773]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:36:53.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:52 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3106503942' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1041454473"}]: dispatch 2024-03-12T03:36:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:52 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1303016910' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/85873921"}]': finished 2024-03-12T03:36:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:52 smithi098 ceph-mon[29473]: osdmap e85: 8 total, 8 up, 8 in 2024-03-12T03:36:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:52 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3106503942' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1041454473"}]: dispatch 2024-03-12T03:36:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:53 smithi153 ceph-mon[25643]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-12T03:36:53.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:53 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3106503942' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1041454473"}]': finished 2024-03-12T03:36:53.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:53 smithi153 ceph-mon[25643]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:36:53.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:53 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1856004018' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4097120357"}]: dispatch 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[24773]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3106503942' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1041454473"}]': finished 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[24773]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1856004018' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4097120357"}]: dispatch 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[29473]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3106503942' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1041454473"}]': finished 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[29473]: osdmap e86: 8 total, 8 up, 8 in 2024-03-12T03:36:54.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:53 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1856004018' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4097120357"}]: dispatch 2024-03-12T03:36:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:54 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1856004018' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4097120357"}]': finished 2024-03-12T03:36:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:54 smithi153 ceph-mon[25643]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:36:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:54 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2656557153' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3533164705"}]: dispatch 2024-03-12T03:36:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:54 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1856004018' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4097120357"}]': finished 2024-03-12T03:36:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:54 smithi098 ceph-mon[24773]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:36:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:54 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2656557153' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3533164705"}]: dispatch 2024-03-12T03:36:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:54 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1856004018' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4097120357"}]': finished 2024-03-12T03:36:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:54 smithi098 ceph-mon[29473]: osdmap e87: 8 total, 8 up, 8 in 2024-03-12T03:36:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:54 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2656557153' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3533164705"}]: dispatch 2024-03-12T03:36:55.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[24773]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2656557153' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3533164705"}]': finished 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[24773]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/223534638' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]: dispatch 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]: dispatch 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[29473]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2656557153' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3533164705"}]': finished 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[29473]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/223534638' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]: dispatch 2024-03-12T03:36:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:55 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]: dispatch 2024-03-12T03:36:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:55 smithi153 ceph-mon[25643]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 79 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-03-12T03:36:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:55 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2656557153' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3533164705"}]': finished 2024-03-12T03:36:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:55 smithi153 ceph-mon[25643]: osdmap e88: 8 total, 8 up, 8 in 2024-03-12T03:36:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:55 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/223534638' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]: dispatch 2024-03-12T03:36:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:55 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]: dispatch 2024-03-12T03:36:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:56 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:56 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]': finished 2024-03-12T03:36:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:56 smithi153 ceph-mon[25643]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:36:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:56 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/132785784' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3827885316"}]: dispatch 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]': finished 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[24773]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/132785784' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3827885316"}]: dispatch 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1016008519"}]': finished 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[29473]: osdmap e89: 8 total, 8 up, 8 in 2024-03-12T03:36:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:56 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/132785784' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3827885316"}]: dispatch 2024-03-12T03:36:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:57 smithi153 ceph-mon[25643]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:57 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/132785784' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3827885316"}]': finished 2024-03-12T03:36:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:57 smithi153 ceph-mon[25643]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:36:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:57 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/4177197207' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]: dispatch 2024-03-12T03:36:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:57 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]: dispatch 2024-03-12T03:36:58.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[24773]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/132785784' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3827885316"}]': finished 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[24773]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/4177197207' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]: dispatch 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]: dispatch 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[29473]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/132785784' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3827885316"}]': finished 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[29473]: osdmap e90: 8 total, 8 up, 8 in 2024-03-12T03:36:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/4177197207' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]: dispatch 2024-03-12T03:36:58.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:57 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]: dispatch 2024-03-12T03:36:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:58 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]': finished 2024-03-12T03:36:58.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:58 smithi153 ceph-mon[25643]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:36:58.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:58 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/945309195' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/2120159776"}]: dispatch 2024-03-12T03:36:58.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:58 smithi153 ceph-mon[25643]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]': finished 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[24773]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/945309195' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/2120159776"}]: dispatch 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[24773]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3603574026"}]': finished 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[29473]: osdmap e91: 8 total, 8 up, 8 in 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/945309195' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/2120159776"}]: dispatch 2024-03-12T03:36:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:58 smithi098 ceph-mon[29473]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:36:59.691 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:36:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:36:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:59 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/945309195' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/2120159776"}]': finished 2024-03-12T03:36:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:59 smithi153 ceph-mon[25643]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:36:59.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:36:59 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/874362346' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/85873921"}]: dispatch 2024-03-12T03:37:00.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:36:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:36:59] "GET /metrics HTTP/1.1" 200 208151 "" "Prometheus/2.33.4" 2024-03-12T03:37:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:59 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/945309195' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/2120159776"}]': finished 2024-03-12T03:37:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:59 smithi098 ceph-mon[24773]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:37:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:36:59 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/874362346' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/85873921"}]: dispatch 2024-03-12T03:37:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:59 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/945309195' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/2120159776"}]': finished 2024-03-12T03:37:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:59 smithi098 ceph-mon[29473]: osdmap e92: 8 total, 8 up, 8 in 2024-03-12T03:37:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:36:59 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/874362346' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/85873921"}]: dispatch 2024-03-12T03:37:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/874362346' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/85873921"}]': finished 2024-03-12T03:37:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[24773]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:37:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/4099068105' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]: dispatch 2024-03-12T03:37:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]: dispatch 2024-03-12T03:37:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[24773]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:01.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/874362346' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/85873921"}]': finished 2024-03-12T03:37:01.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[29473]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:37:01.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/4099068105' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]: dispatch 2024-03-12T03:37:01.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]: dispatch 2024-03-12T03:37:01.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:00 smithi098 ceph-mon[29473]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:01.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:00 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/874362346' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/85873921"}]': finished 2024-03-12T03:37:01.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:00 smithi153 ceph-mon[25643]: osdmap e93: 8 total, 8 up, 8 in 2024-03-12T03:37:01.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:00 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/4099068105' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]: dispatch 2024-03-12T03:37:01.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:00 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]: dispatch 2024-03-12T03:37:01.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:00 smithi153 ceph-mon[25643]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:01.977 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:01 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]': finished 2024-03-12T03:37:01.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:01 smithi098 ceph-mon[24773]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:37:01.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:01 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1018136161' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/2120159776"}]: dispatch 2024-03-12T03:37:01.978 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:01 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]': finished 2024-03-12T03:37:01.978 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:01 smithi098 ceph-mon[29473]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:37:01.978 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:01 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1018136161' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/2120159776"}]: dispatch 2024-03-12T03:37:02.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:01 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3824965392"}]': finished 2024-03-12T03:37:02.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:01 smithi153 ceph-mon[25643]: osdmap e94: 8 total, 8 up, 8 in 2024-03-12T03:37:02.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:01 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1018136161' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/2120159776"}]: dispatch 2024-03-12T03:37:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:02 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1018136161' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/2120159776"}]': finished 2024-03-12T03:37:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:02 smithi098 ceph-mon[24773]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:37:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:02 smithi098 ceph-mon[24773]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:02 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1018136161' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/2120159776"}]': finished 2024-03-12T03:37:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:02 smithi098 ceph-mon[29473]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:37:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:02 smithi098 ceph-mon[29473]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:02 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1018136161' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/2120159776"}]': finished 2024-03-12T03:37:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:02 smithi153 ceph-mon[25643]: osdmap e95: 8 total, 8 up, 8 in 2024-03-12T03:37:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:02 smithi153 ceph-mon[25643]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:03.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:37:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:37:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:37:03.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:37:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:37:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:37:05.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:05 smithi098 ceph-mon[24773]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-12T03:37:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:05 smithi098 ceph-mon[29473]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-12T03:37:05.896 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:05 smithi153 ceph-mon[25643]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-12T03:37:06.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:06 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:06 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:06.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:06 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:07 smithi098 ceph-mon[24773]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:37:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:07 smithi098 ceph-mon[29473]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:37:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:07 smithi153 ceph-mon[25643]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:37:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:09 smithi153 ceph-mon[25643]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:09.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:37:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:37:09.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:09 smithi098 ceph-mon[29473]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:09 smithi098 ceph-mon[24773]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:09.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:37:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:09] "GET /metrics HTTP/1.1" 200 208151 "" "Prometheus/2.33.4" 2024-03-12T03:37:11.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:11 smithi098 ceph-mon[24773]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:37:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:11 smithi098 ceph-mon[29473]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:37:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:11 smithi153 ceph-mon[25643]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T03:37:13.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:13 smithi098 ceph-mon[24773]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-12T03:37:13.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:13 smithi098 ceph-mon[29473]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-12T03:37:13.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:37:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:37:13.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:37:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:13 smithi153 ceph-mon[25643]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-12T03:37:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:15 smithi098 ceph-mon[24773]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:15 smithi098 ceph-mon[29473]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:15 smithi153 ceph-mon[25643]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:16.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:16 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:16 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:16.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:16 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:17 smithi098 ceph-mon[24773]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:17.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:17 smithi098 ceph-mon[29473]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:17 smithi153 ceph-mon[25643]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:19 smithi153 ceph-mon[25643]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:19.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:37:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:37:19.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:19 smithi098 ceph-mon[29473]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:19 smithi098 ceph-mon[24773]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:19.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:37:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:19] "GET /metrics HTTP/1.1" 200 208148 "" "Prometheus/2.33.4" 2024-03-12T03:37:21.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:21 smithi098 ceph-mon[24773]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:21.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:21 smithi098 ceph-mon[29473]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:21 smithi153 ceph-mon[25643]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:23.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:23 smithi098 ceph-mon[24773]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:23.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:23 smithi098 ceph-mon[29473]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:23.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:23 smithi153 ceph-mon[25643]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:25.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:25 smithi098 ceph-mon[24773]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:25.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:25 smithi098 ceph-mon[29473]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:25 smithi153 ceph-mon[25643]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:26 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:26 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:26 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:27 smithi098 ceph-mon[24773]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:27 smithi098 ceph-mon[29473]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:27 smithi153 ceph-mon[25643]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:29 smithi153 ceph-mon[25643]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:29.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:37:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:37:29.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:29 smithi098 ceph-mon[24773]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:29 smithi098 ceph-mon[29473]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:29.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:37:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:29] "GET /metrics HTTP/1.1" 200 208080 "" "Prometheus/2.33.4" 2024-03-12T03:37:31.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:31 smithi098 ceph-mon[24773]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:31.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:31 smithi098 ceph-mon[29473]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:31.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:31 smithi153 ceph-mon[25643]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:33 smithi098 ceph-mon[24773]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:33 smithi098 ceph-mon[29473]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:33 smithi153 ceph-mon[25643]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:35.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:35 smithi098 ceph-mon[24773]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:35.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:35 smithi098 ceph-mon[29473]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:35 smithi153 ceph-mon[25643]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:36.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:36 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:36.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:36 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:36.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:36 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:37.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:37 smithi098 ceph-mon[24773]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:37.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:37 smithi098 ceph-mon[29473]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:37 smithi153 ceph-mon[25643]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:39.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:39 smithi153 ceph-mon[25643]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:39.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:37:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:37:39.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:39 smithi098 ceph-mon[24773]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:39 smithi098 ceph-mon[29473]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:39.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:37:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:39] "GET /metrics HTTP/1.1" 200 208080 "" "Prometheus/2.33.4" 2024-03-12T03:37:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:37:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:37:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:37:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:37:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:37:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:37:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:37:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:37:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:37:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:37:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:37:41.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:37:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:41 smithi153 ceph-mon[25643]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:42.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:41 smithi098 ceph-mon[24773]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:42.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:41 smithi098 ceph-mon[29473]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:43 smithi153 ceph-mon[25643]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:43 smithi098 ceph-mon[24773]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:44.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:43 smithi098 ceph-mon[29473]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:45.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:45 smithi153 ceph-mon[25643]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:45 smithi098 ceph-mon[24773]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:46.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:45 smithi098 ceph-mon[29473]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:46.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:46 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:46 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:46 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:47 smithi153 ceph-mon[25643]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:48.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:47 smithi098 ceph-mon[24773]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:47 smithi098 ceph-mon[29473]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:49.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:49 smithi153 ceph-mon[25643]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:49.664 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:37:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:37:49.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:49 smithi098 ceph-mon[24773]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:49.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:49 smithi098 ceph-mon[29473]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:49.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:37:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:49] "GET /metrics HTTP/1.1" 200 208043 "" "Prometheus/2.33.4" 2024-03-12T03:37:50.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:50 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:37:50.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:50 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:37:50.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:50 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:37:50.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:50 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:37:50.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:50 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:37:50.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:50 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:37:50.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:50 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:37:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:50 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:37:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:50 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:37:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:51 smithi098 ceph-mon[24773]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:51.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:51 smithi098 ceph-mon[29473]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:51 smithi153 ceph-mon[25643]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:53 smithi098 ceph-mon[24773]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:53 smithi098 ceph-mon[29473]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:53 smithi153 ceph-mon[25643]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:55 smithi098 ceph-mon[24773]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:55 smithi098 ceph-mon[29473]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:55 smithi153 ceph-mon[25643]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:56.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:56 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:56 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:56 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:37:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:57 smithi098 ceph-mon[24773]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:57.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:57 smithi098 ceph-mon[29473]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:57 smithi153 ceph-mon[25643]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:37:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:37:59 smithi153 ceph-mon[25643]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:59.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:37:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:37:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:37:59 smithi098 ceph-mon[24773]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:37:59 smithi098 ceph-mon[29473]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:37:59.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:37:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:37:59] "GET /metrics HTTP/1.1" 200 208049 "" "Prometheus/2.33.4" 2024-03-12T03:38:01.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:01 smithi098 ceph-mon[29473]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:01 smithi098 ceph-mon[24773]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:01 smithi153 ceph-mon[25643]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:03.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:03 smithi098 ceph-mon[24773]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:03 smithi098 ceph-mon[29473]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:03 smithi153 ceph-mon[25643]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:05.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:05 smithi098 ceph-mon[24773]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:05 smithi098 ceph-mon[29473]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:05 smithi153 ceph-mon[25643]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:06 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:06 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:06.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:06 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:07 smithi098 ceph-mon[24773]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:07 smithi098 ceph-mon[29473]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:07 smithi153 ceph-mon[25643]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:09 smithi153 ceph-mon[25643]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:09.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:38:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:38:09.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:09 smithi098 ceph-mon[24773]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:09 smithi098 ceph-mon[29473]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:09.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:38:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:09] "GET /metrics HTTP/1.1" 200 208049 "" "Prometheus/2.33.4" 2024-03-12T03:38:11.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:11 smithi098 ceph-mon[24773]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:11 smithi098 ceph-mon[29473]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:11 smithi153 ceph-mon[25643]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:13.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:13 smithi098 ceph-mon[29473]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:13.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:13 smithi098 ceph-mon[24773]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:13 smithi153 ceph-mon[25643]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:15 smithi098 ceph-mon[24773]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:15 smithi098 ceph-mon[29473]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:15 smithi153 ceph-mon[25643]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:16 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:16.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:16 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:16.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:16 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:17 smithi153 ceph-mon[25643]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:18.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:17 smithi098 ceph-mon[24773]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:17 smithi098 ceph-mon[29473]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:19 smithi153 ceph-mon[25643]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:19.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:38:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:38:19.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:19 smithi098 ceph-mon[29473]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:19 smithi098 ceph-mon[24773]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:19.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:38:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:19] "GET /metrics HTTP/1.1" 200 208056 "" "Prometheus/2.33.4" 2024-03-12T03:38:21.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:21 smithi098 ceph-mon[24773]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:21.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:21 smithi098 ceph-mon[29473]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:21 smithi153 ceph-mon[25643]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:23.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:23 smithi098 ceph-mon[24773]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:23.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:23 smithi098 ceph-mon[29473]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:23 smithi153 ceph-mon[25643]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:25.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:25 smithi098 ceph-mon[24773]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:25.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:25 smithi098 ceph-mon[29473]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:25 smithi153 ceph-mon[25643]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:26 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:26 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:26 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:27 smithi153 ceph-mon[25643]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:28.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:27 smithi098 ceph-mon[24773]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:27 smithi098 ceph-mon[29473]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:29 smithi153 ceph-mon[25643]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:29.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:38:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:38:29.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:29 smithi098 ceph-mon[24773]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:29 smithi098 ceph-mon[29473]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:29.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:38:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:29] "GET /metrics HTTP/1.1" 200 208059 "" "Prometheus/2.33.4" 2024-03-12T03:38:31.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:31 smithi098 ceph-mon[24773]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:31.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:31 smithi098 ceph-mon[29473]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:31.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:31 smithi153 ceph-mon[25643]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:33 smithi098 ceph-mon[24773]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:33 smithi098 ceph-mon[29473]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:33 smithi153 ceph-mon[25643]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:35 smithi153 ceph-mon[25643]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:36.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:35 smithi098 ceph-mon[24773]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:35 smithi098 ceph-mon[29473]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:36.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:36 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:36 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:36 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:37 smithi153 ceph-mon[25643]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:37 smithi098 ceph-mon[24773]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:37 smithi098 ceph-mon[29473]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:39 smithi153 ceph-mon[25643]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:39.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:38:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:38:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:39 smithi098 ceph-mon[24773]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:39.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:38:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:39] "GET /metrics HTTP/1.1" 200 208059 "" "Prometheus/2.33.4" 2024-03-12T03:38:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:39 smithi098 ceph-mon[29473]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:38:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:38:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:38:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:38:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:38:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:38:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:38:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:38:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:38:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:38:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:38:41.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:38:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:41 smithi153 ceph-mon[25643]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:42.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:41 smithi098 ceph-mon[24773]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:42.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:41 smithi098 ceph-mon[29473]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:43 smithi153 ceph-mon[25643]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:43 smithi098 ceph-mon[24773]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:44.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:43 smithi098 ceph-mon[29473]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:45.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:45 smithi153 ceph-mon[25643]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:45 smithi098 ceph-mon[24773]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:46.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:45 smithi098 ceph-mon[29473]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:46.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:46 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:46 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:46 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:47 smithi153 ceph-mon[25643]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:48.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:47 smithi098 ceph-mon[24773]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:47 smithi098 ceph-mon[29473]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:49 smithi153 ceph-mon[25643]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:49.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:38:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:38:49.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:49 smithi098 ceph-mon[24773]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:49.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:49 smithi098 ceph-mon[29473]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:49.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:38:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:49] "GET /metrics HTTP/1.1" 200 208056 "" "Prometheus/2.33.4" 2024-03-12T03:38:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:50 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:50 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:50 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:38:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:50 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:50 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:50 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:38:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:50 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:38:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:50 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:38:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:50 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:38:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:51 smithi098 ceph-mon[24773]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:51.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:51 smithi098 ceph-mon[29473]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:51 smithi153 ceph-mon[25643]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:53 smithi153 ceph-mon[25643]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:54.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:53 smithi098 ceph-mon[29473]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:54.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:53 smithi098 ceph-mon[24773]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:55 smithi153 ceph-mon[25643]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:56.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:55 smithi098 ceph-mon[24773]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:56.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:55 smithi098 ceph-mon[29473]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:56 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:56 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:56 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:38:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:57 smithi153 ceph-mon[25643]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:57 smithi098 ceph-mon[24773]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:57 smithi098 ceph-mon[29473]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:38:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:38:59 smithi153 ceph-mon[25643]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:59.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:38:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:38:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:38:59 smithi098 ceph-mon[24773]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:38:59 smithi098 ceph-mon[29473]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:38:59.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:38:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:38:59] "GET /metrics HTTP/1.1" 200 208042 "" "Prometheus/2.33.4" 2024-03-12T03:39:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:01 smithi153 ceph-mon[25643]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:02.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:01 smithi098 ceph-mon[24773]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:02.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:01 smithi098 ceph-mon[29473]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:03 smithi153 ceph-mon[25643]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:04.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:03 smithi098 ceph-mon[24773]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:04.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:03 smithi098 ceph-mon[29473]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:05 smithi153 ceph-mon[25643]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:06.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:05 smithi098 ceph-mon[24773]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:05 smithi098 ceph-mon[29473]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:06.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:06 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:06 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:06 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:07 smithi153 ceph-mon[25643]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:08.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:07 smithi098 ceph-mon[24773]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:07 smithi098 ceph-mon[29473]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:09 smithi153 ceph-mon[25643]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:09.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:39:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:39:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:09 smithi098 ceph-mon[24773]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:09 smithi098 ceph-mon[29473]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:09.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:39:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:09] "GET /metrics HTTP/1.1" 200 208042 "" "Prometheus/2.33.4" 2024-03-12T03:39:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:11 smithi153 ceph-mon[25643]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:12.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:11 smithi098 ceph-mon[24773]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:12.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:11 smithi098 ceph-mon[29473]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:13 smithi153 ceph-mon[25643]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:13 smithi098 ceph-mon[24773]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:13 smithi098 ceph-mon[29473]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:15 smithi153 ceph-mon[25643]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:16.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:15 smithi098 ceph-mon[24773]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:15 smithi098 ceph-mon[29473]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:16.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:16 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:17.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:16 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:16 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:17 smithi153 ceph-mon[25643]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:18.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:17 smithi098 ceph-mon[24773]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:17 smithi098 ceph-mon[29473]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:19 smithi153 ceph-mon[25643]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:19.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:39:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:39:19.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:19 smithi098 ceph-mon[24773]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:19.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:19 smithi098 ceph-mon[29473]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:19.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:39:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:19] "GET /metrics HTTP/1.1" 200 208010 "" "Prometheus/2.33.4" 2024-03-12T03:39:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:21 smithi153 ceph-mon[25643]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:22.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:21 smithi098 ceph-mon[24773]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:21 smithi098 ceph-mon[29473]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:23 smithi153 ceph-mon[25643]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:24.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:23 smithi098 ceph-mon[24773]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:24.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:23 smithi098 ceph-mon[29473]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:25 smithi153 ceph-mon[25643]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:26.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:25 smithi098 ceph-mon[24773]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:26.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:25 smithi098 ceph-mon[29473]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:26 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:27.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:26 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:27.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:26 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:27 smithi153 ceph-mon[25643]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:27 smithi098 ceph-mon[24773]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:27 smithi098 ceph-mon[29473]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:29 smithi153 ceph-mon[25643]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:29.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:39:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:39:29.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:29 smithi098 ceph-mon[24773]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:29 smithi098 ceph-mon[29473]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:29.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:39:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:29] "GET /metrics HTTP/1.1" 200 207999 "" "Prometheus/2.33.4" 2024-03-12T03:39:31.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:31 smithi153 ceph-mon[25643]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:32.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:31 smithi098 ceph-mon[24773]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:32.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:31 smithi098 ceph-mon[29473]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:33 smithi153 ceph-mon[25643]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:34.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:33 smithi098 ceph-mon[24773]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:33 smithi098 ceph-mon[29473]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:35 smithi153 ceph-mon[25643]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:36.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:35 smithi098 ceph-mon[24773]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:35 smithi098 ceph-mon[29473]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:36.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:36 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:36.722 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:36 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:36.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:36 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:37.117 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-12T03:39:37.734 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:37 smithi098 ceph-mon[24773]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:37.735 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:37 smithi098 ceph-mon[29473]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:37 smithi153 ceph-mon[25643]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:39.427 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-12T03:39:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:39 smithi153 ceph-mon[25643]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:39.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:39:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:39:39.739 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:39 smithi098 ceph-mon[24773]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:39.739 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:39 smithi098 ceph-mon[29473]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:39.740 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:39:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:39] "GET /metrics HTTP/1.1" 200 207999 "" "Prometheus/2.33.4" 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:39:40.880 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:39:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:39:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:39:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:39:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:39:41.901 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:41 smithi098 ceph-mon[24773]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:41.901 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:41 smithi098 ceph-mon[29473]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:41.925 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-12T03:39:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:41 smithi153 ceph-mon[25643]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:43 smithi153 ceph-mon[25643]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:43 smithi098 ceph-mon[24773]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:44.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:43 smithi098 ceph-mon[29473]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:44.423 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T03:39:45.847 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:45 smithi098 ceph-mon[24773]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:45.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:45 smithi098 ceph-mon[29473]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:45.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:45 smithi153 ceph-mon[25643]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (3m) 2m ago 3m 15.4M - ba2b418f427c c12f6c606e5f 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (3m) 2m ago 3m 48.9M - 8.3.5 dad864ee21e9 7864a9bab9e4 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (3m) 2m ago 3m 71.1M - 3.5 e1d6a67b021e 4c7a21937fd9 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443 running (8m) 2m ago 8m 410M - 17.2.0 e1d6a67b021e d93cdaa465f5 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:9283 running (10m) 2m ago 10m 454M - 17.2.0 e1d6a67b021e 3c66c3a9135b 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (10m) 2m ago 10m 46.2M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (8m) 2m ago 8m 39.0M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (8m) 2m ago 8m 39.7M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (4m) 2m ago 4m 17.8M - 1dbe0e931976 fd51a335b42c 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (4m) 2m ago 4m 16.4M - 1dbe0e931976 270f07cc8db8 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (8m) 2m ago 8m 50.5M 4016M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:39:46.252 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (7m) 2m ago 7m 51.6M 4016M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (7m) 2m ago 7m 44.8M 4016M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (6m) 2m ago 6m 45.9M 4016M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (6m) 2m ago 6m 49.6M 4272M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (6m) 2m ago 6m 47.2M 4272M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (5m) 2m ago 5m 44.0M 4272M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (5m) 2m ago 5m 46.3M 4272M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (3m) 2m ago 4m 44.1M - 514e6a882f6e 7edcfd1725dd 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (3m) 2m ago 3m 69.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:39:46.253 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (3m) 2m ago 3m 72.3M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:39:46.808 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions' 2024-03-12T03:39:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:47 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:47 smithi098 ceph-mon[29473]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:47 smithi098 ceph-mon[29473]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:47 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:47 smithi098 ceph-mon[24773]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:47.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:47 smithi098 ceph-mon[24773]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:47 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:47 smithi153 ceph-mon[25643]: from='client.24874 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:47 smithi153 ceph-mon[25643]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:39:48.748 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:39:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:39:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:39:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:39:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:39:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-03-12T03:39:48.749 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:39:48.749 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:39:49.451 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph -s' 2024-03-12T03:39:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:49 smithi153 ceph-mon[25643]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:49 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1367181294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:39:49.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:39:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:39:49.762 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:39:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:49] "GET /metrics HTTP/1.1" 200 207996 "" "Prometheus/2.33.4" 2024-03-12T03:39:49.763 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:49 smithi098 ceph-mon[29473]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:49.763 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:49 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1367181294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:39:49.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:49 smithi098 ceph-mon[24773]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:49.763 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:49 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1367181294' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:39:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:50 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:50 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:50 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:50 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:50.601 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:50 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:50.601 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:50 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:51.495 INFO:teuthology.orchestra.run.smithi098.stdout: cluster: 2024-03-12T03:39:51.495 INFO:teuthology.orchestra.run.smithi098.stdout: id: ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:39:51.495 INFO:teuthology.orchestra.run.smithi098.stdout: health: HEALTH_OK 2024-03-12T03:39:51.495 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:39:51.495 INFO:teuthology.orchestra.run.smithi098.stdout: services: 2024-03-12T03:39:51.495 INFO:teuthology.orchestra.run.smithi098.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-03-12T03:39:51.495 INFO:teuthology.orchestra.run.smithi098.stdout: mgr: y(active, since 5m), standbys: x 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: data: 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: pools: 6 pools, 161 pgs 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: objects: 241 objects, 585 KiB 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: usage: 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: pgs: 161 active+clean 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: io: 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:39:51.496 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:39:51.788 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:51 smithi098 ceph-mon[24773]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:51.788 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:51 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:51.788 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:51 smithi098 ceph-mon[29473]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:51.789 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:51 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:51 smithi153 ceph-mon[25643]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:51.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:51 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:52.079 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ls' 2024-03-12T03:39:52.717 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:52 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/183030046' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:39:52.717 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:52 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/183030046' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:39:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:52 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/183030046' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:39:53.592 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:53 smithi098 ceph-mon[24773]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:53.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:53 smithi098 ceph-mon[29473]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:53 smithi153 ceph-mon[25643]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:54.003 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi098=a;count:1 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:grafana ?:3000 1/1 3m ago 5m smithi153=a;count:1 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 2/2 3m ago 8m smithi098=y;smithi153=x;count:2 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 3m ago 9m smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b;count:3 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi098=a;smithi153=b;count:2 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:osd 8 3m ago - 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus ?:9095 1/1 3m ago 5m smithi153=a;count:1 2024-03-12T03:39:54.004 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-03-12T03:39:54.617 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-12T03:39:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:55 smithi153 ceph-mon[25643]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:55 smithi153 ceph-mon[25643]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:55.959 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:55 smithi098 ceph-mon[24773]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:55.960 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:55 smithi098 ceph-mon[24773]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:55.960 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:55 smithi098 ceph-mon[29473]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:55.960 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:55 smithi098 ceph-mon[29473]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:56.935 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled to redeploy mgr.x on host 'smithi153' 2024-03-12T03:39:57.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps --refresh' 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2445804747' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:39:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2445804747' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:39:57.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:57 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2445804747' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:39:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:57 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:39:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:58 smithi153 ceph-mon[25643]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:58 smithi153 ceph-mon[25643]: Schedule redeploy daemon mgr.x 2024-03-12T03:39:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:58 smithi153 ceph-mon[25643]: Deploying daemon mgr.x on smithi153 2024-03-12T03:39:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:58 smithi098 ceph-mon[24773]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:58 smithi098 ceph-mon[24773]: Schedule redeploy daemon mgr.x 2024-03-12T03:39:58.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:58 smithi098 ceph-mon[24773]: Deploying daemon mgr.x on smithi153 2024-03-12T03:39:59.000 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:58 smithi098 ceph-mon[29473]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:39:59.000 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:58 smithi098 ceph-mon[29473]: Schedule redeploy daemon mgr.x 2024-03-12T03:39:59.000 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:58 smithi098 ceph-mon[29473]: Deploying daemon mgr.x on smithi153 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (3m) 3m ago 4m 15.4M - ba2b418f427c c12f6c606e5f 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (3m) 3m ago 3m 48.9M - 8.3.5 dad864ee21e9 7864a9bab9e4 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (3m) 3m ago 3m 71.1M - 3.5 e1d6a67b021e 4c7a21937fd9 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443 running (8m) 3m ago 8m 410M - 17.2.0 e1d6a67b021e d93cdaa465f5 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:9283 running (10m) 3m ago 10m 454M - 17.2.0 e1d6a67b021e 3c66c3a9135b 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (10m) 3m ago 10m 46.2M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (8m) 3m ago 8m 39.0M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:39:59.619 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (9m) 3m ago 9m 39.7M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (4m) 3m ago 4m 17.8M - 1dbe0e931976 fd51a335b42c 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (4m) 3m ago 4m 16.4M - 1dbe0e931976 270f07cc8db8 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (8m) 3m ago 8m 50.5M 4016M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (7m) 3m ago 7m 51.6M 4016M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (7m) 3m ago 7m 44.8M 4016M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (7m) 3m ago 7m 45.9M 4016M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (6m) 3m ago 6m 49.6M 4272M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (6m) 3m ago 6m 47.2M 4272M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (6m) 3m ago 6m 44.0M 4272M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (5m) 3m ago 5m 46.3M 4272M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (3m) 3m ago 5m 44.1M - 514e6a882f6e 7edcfd1725dd 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (3m) 3m ago 3m 69.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:39:59.620 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (3m) 3m ago 3m 72.3M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:39:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:39:59 smithi153 ceph-mon[25643]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:39:59.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:39:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:40:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:39:59 smithi098 ceph-mon[24773]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:39:59 smithi098 ceph-mon[29473]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:00.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:39:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:39:59] "GET /metrics HTTP/1.1" 200 207993 "" "Prometheus/2.33.4" 2024-03-12T03:40:00.212 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'sleep 180' 2024-03-12T03:40:00.806 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:00 smithi098 ceph-mon[24773]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:40:00.807 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:00 smithi098 ceph-mon[24773]: overall HEALTH_OK 2024-03-12T03:40:00.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:00 smithi098 ceph-mon[29473]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:40:00.807 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:00 smithi098 ceph-mon[29473]: overall HEALTH_OK 2024-03-12T03:40:00.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:00 smithi153 ceph-mon[25643]: from='client.24910 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:40:00.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:00 smithi153 ceph-mon[25643]: overall HEALTH_OK 2024-03-12T03:40:01.936 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:01 smithi098 ceph-mon[24773]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:01.937 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:01 smithi098 ceph-mon[29473]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:01 smithi153 ceph-mon[25643]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:03.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:03 smithi098 ceph-mon[24773]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:03 smithi098 ceph-mon[29473]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:03 smithi153 ceph-mon[25643]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:05 smithi153 ceph-mon[25643]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:06.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:05 smithi098 ceph-mon[24773]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:05 smithi098 ceph-mon[29473]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:07 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:07 smithi153 ceph-mon[25643]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:08.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:07 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:08.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:07 smithi098 ceph-mon[24773]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:07 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:07 smithi098 ceph-mon[29473]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:09 smithi153 ceph-mon[25643]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:09.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:40:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:09 smithi098 ceph-mon[24773]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:09.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:40:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:09] "GET /metrics HTTP/1.1" 200 207993 "" "Prometheus/2.33.4" 2024-03-12T03:40:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:09 smithi098 ceph-mon[29473]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:11 smithi153 ceph-mon[25643]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:12.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:11 smithi098 ceph-mon[24773]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:12.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:11 smithi098 ceph-mon[29473]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:13 smithi153 ceph-mon[25643]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:13 smithi098 ceph-mon[24773]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:13 smithi098 ceph-mon[29473]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:15 smithi153 ceph-mon[25643]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:16.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:15 smithi098 ceph-mon[24773]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:15 smithi098 ceph-mon[29473]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:17 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:17 smithi153 ceph-mon[25643]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:17 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:17 smithi098 ceph-mon[24773]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:17 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:17 smithi098 ceph-mon[29473]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:19 smithi153 ceph-mon[25643]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:19.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[26821]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:40:19.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:19 smithi098 ceph-mon[24773]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:19.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:40:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:19] "GET /metrics HTTP/1.1" 200 207991 "" "Prometheus/2.33.4" 2024-03-12T03:40:19.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:19 smithi098 ceph-mon[29473]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:21 smithi153 ceph-mon[25643]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:21 smithi098 ceph-mon[24773]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:21 smithi098 ceph-mon[29473]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:23.816 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:23 smithi153 ceph-mon[25643]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:24.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:23 smithi098 ceph-mon[24773]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:24.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:23 smithi098 ceph-mon[29473]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:24.828 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:24 smithi153 systemd[1]: Stopping Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:40:24.828 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:24 smithi153 bash[45226]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr.x" found: no such container 2024-03-12T03:40:25.099 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:24 smithi153 podman[45239]: 2024-03-12 03:40:24.826280796 +0000 UTC m=+0.129498284 container died d93cdaa465f5033f86ffe6a73355d35eb7b609905fe93051dfa4d363cebb6194 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, version=8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=) 2024-03-12T03:40:25.375 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 podman[45239]: 2024-03-12 03:40:25.171819537 +0000 UTC m=+0.475037052 container cleanup d93cdaa465f5033f86ffe6a73355d35eb7b609905fe93051dfa4d363cebb6194 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_CLEAN=True, architecture=x86_64) 2024-03-12T03:40:25.376 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 bash[45239]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x 2024-03-12T03:40:25.376 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 podman[45253]: 2024-03-12 03:40:25.339520737 +0000 UTC m=+0.506694861 container remove d93cdaa465f5033f86ffe6a73355d35eb7b609905fe93051dfa4d363cebb6194 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, architecture=x86_64, ceph=True, io.openshift.expose-services=, vcs-type=git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD) 2024-03-12T03:40:25.376 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:40:25.376 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 bash[45266]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr.x" found: no such container 2024-03-12T03:40:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:25 smithi153 ceph-mon[25643]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:25.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-12T03:40:25.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 systemd[1]: Stopped Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:40:25.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service: Consumed 16.822s CPU time. 2024-03-12T03:40:25.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 systemd[1]: Starting Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:40:26.043 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 podman[45375]: 2024-03-12 03:40:25.774349763 +0000 UTC m=+0.093788651 container create 2e7f2996e113d35dca20c038327a90364ed2a1d144060493cb1c05c54a0b6213 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-12T03:40:26.043 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 podman[45375]: 2024-03-12 03:40:25.699293859 +0000 UTC m=+0.018732748 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:40:26.043 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 podman[45375]: 2024-03-12 03:40:25.960184961 +0000 UTC m=+0.279623860 container init 2e7f2996e113d35dca20c038327a90364ed2a1d144060493cb1c05c54a0b6213 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-12T03:40:26.043 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 podman[45375]: 2024-03-12 03:40:25.962736101 +0000 UTC m=+0.282174994 container start 2e7f2996e113d35dca20c038327a90364ed2a1d144060493cb1c05c54a0b6213 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T03:40:26.043 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:25 smithi153 ceph-mgr[45394]: -- 172.21.15.153:0/649841357 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56358004e5a0 con 0x563580022800 2024-03-12T03:40:26.044 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:26 smithi153 bash[45375]: 2e7f2996e113d35dca20c038327a90364ed2a1d144060493cb1c05c54a0b6213 2024-03-12T03:40:26.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:25 smithi098 ceph-mon[24773]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:26.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:25 smithi098 ceph-mon[29473]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:26.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:26 smithi153 systemd[1]: Started Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:40:26.825 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:26.549+0000 7f05dfd1f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:40:26.825 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:26.641+0000 7f05dfd1f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:40:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[24773]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:27 smithi098 ceph-mon[29473]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:27.368 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:27.327+0000 7f05dfd1f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:40:27.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:27.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:40:27.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:40:27.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:27.369 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-mon[25643]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:27.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:27.428+0000 7f05dfd1f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:40:27.627 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:27.584+0000 7f05dfd1f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:40:27.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:27.686+0000 7f05dfd1f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:40:27.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:27.781+0000 7f05dfd1f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:40:28.346 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:28.085+0000 7f05dfd1f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:40:28.347 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:28.200+0000 7f05dfd1f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:40:28.347 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:28.299+0000 7f05dfd1f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:40:28.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:28.397+0000 7f05dfd1f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:40:28.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:28.578+0000 7f05dfd1f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:40:29.741 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:29 smithi153 ceph-mon[25643]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:29.741 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:29.600+0000 7f05dfd1f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:40:29.741 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:29.738+0000 7f05dfd1f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:40:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:29 smithi098 ceph-mon[24773]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:29 smithi098 ceph-mon[29473]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:29.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:40:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:29] "GET /metrics HTTP/1.1" 200 207998 "" "Prometheus/2.33.4" 2024-03-12T03:40:30.380 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:30.086+0000 7f05dfd1f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:40:30.380 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:30.180+0000 7f05dfd1f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:40:30.380 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:30.377+0000 7f05dfd1f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:40:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:30.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:30.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:30.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:30.559+0000 7f05dfd1f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:40:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:30 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:30 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:30 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:30 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:30 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:30 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:40:31.112 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:30.846+0000 7f05dfd1f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:40:31.112 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:31.109+0000 7f05dfd1f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:40:31.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:31.349+0000 7f05dfd1f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:40:31.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:31 smithi153 ceph-mon[25643]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:31.974 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:31 smithi098 ceph-mon[24773]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:31.975 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:31 smithi098 ceph-mon[29473]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:32.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:32.157+0000 7f05dfd1f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:40:33.046 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:32.728+0000 7f05dfd1f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:40:33.046 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:32.827+0000 7f05dfd1f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:40:33.046 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:33.043+0000 7f05dfd1f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:40:33.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:40:33] ENGINE Bus STARTING 2024-03-12T03:40:33.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: CherryPy Checker: 2024-03-12T03:40:33.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: The Application mounted at '' has an empty config. 2024-03-12T03:40:33.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:40:33.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:40:33] ENGINE Serving on http://:::9283 2024-03-12T03:40:33.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:40:33] ENGINE Bus STARTED 2024-03-12T03:40:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-mon[25643]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-mon[25643]: Standby manager daemon x restarted 2024-03-12T03:40:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-mon[25643]: Standby manager daemon x started 2024-03-12T03:40:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:40:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:40:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:40:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:33 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:40:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[29473]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[29473]: Standby manager daemon x restarted 2024-03-12T03:40:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[29473]: Standby manager daemon x started 2024-03-12T03:40:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:40:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:40:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:40:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:40:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[24773]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[24773]: Standby manager daemon x restarted 2024-03-12T03:40:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[24773]: Standby manager daemon x started 2024-03-12T03:40:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:40:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:40:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:40:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:33 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:40:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:34 smithi153 ceph-mon[25643]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-12T03:40:35.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:34 smithi098 ceph-mon[24773]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-12T03:40:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:34 smithi098 ceph-mon[29473]: mgrmap e21: y(active, since 5m), standbys: x 2024-03-12T03:40:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:35 smithi153 ceph-mon[25643]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:36.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:35 smithi098 ceph-mon[24773]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:35 smithi098 ceph-mon[29473]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:37 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:37 smithi153 ceph-mon[25643]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:37 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:37 smithi098 ceph-mon[24773]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:37 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:37 smithi098 ceph-mon[29473]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:39 smithi153 ceph-mon[25643]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:39.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:40:39.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:39 smithi098 ceph-mon[24773]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:39 smithi098 ceph-mon[29473]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:39.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:40:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:39] "GET /metrics HTTP/1.1" 200 207998 "" "Prometheus/2.33.4" 2024-03-12T03:40:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:40:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:40:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:40:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:40:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:40:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:40:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:40:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:40:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:40:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:40:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:40:41.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:40:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:41 smithi153 ceph-mon[25643]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:42.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:41 smithi098 ceph-mon[24773]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:42.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:41 smithi098 ceph-mon[29473]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:43 smithi098 ceph-mon[24773]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:43 smithi098 ceph-mon[29473]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:43.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:40:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:40:43.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:40:43.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:40:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:40:43.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:40:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:43 smithi153 ceph-mon[25643]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:45.891 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:45 smithi153 ceph-mon[25643]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:45 smithi098 ceph-mon[24773]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:46.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:45 smithi098 ceph-mon[29473]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:47 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:47 smithi153 ceph-mon[25643]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:48.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:47 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:48.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:47 smithi098 ceph-mon[24773]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:47 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:47 smithi098 ceph-mon[29473]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:49 smithi153 ceph-mon[25643]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:49.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:40:49.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:49 smithi098 ceph-mon[24773]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:49.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:49 smithi098 ceph-mon[29473]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:49.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:40:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:49] "GET /metrics HTTP/1.1" 200 208013 "" "Prometheus/2.33.4" 2024-03-12T03:40:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:51 smithi153 ceph-mon[25643]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:51 smithi098 ceph-mon[29473]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:52.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:51 smithi098 ceph-mon[24773]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:53.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:53 smithi098 ceph-mon[29473]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:53 smithi098 ceph-mon[24773]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:40:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:40:53.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:40:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:40:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:40:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:40:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:40:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:40:53.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:40:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:53 smithi153 ceph-mon[25643]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:55 smithi153 ceph-mon[25643]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:56.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:55 smithi098 ceph-mon[24773]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:56.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:55 smithi098 ceph-mon[29473]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:57 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:57 smithi153 ceph-mon[25643]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:57 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:57 smithi098 ceph-mon[24773]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:57 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:40:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:57 smithi098 ceph-mon[29473]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:40:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:40:59 smithi153 ceph-mon[25643]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:59.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:40:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:40:59.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:40:59 smithi098 ceph-mon[24773]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:40:59 smithi098 ceph-mon[29473]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:40:59.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:40:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:40:59] "GET /metrics HTTP/1.1" 200 208008 "" "Prometheus/2.33.4" 2024-03-12T03:41:01.876 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:01 smithi098 ceph-mon[24773]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:01.877 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:01 smithi098 ceph-mon[29473]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:01.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:01 smithi153 ceph-mon[25643]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:03.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:03 smithi098 ceph-mon[24773]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:03 smithi098 ceph-mon[29473]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:03.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:41:03.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:03.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:03.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:41:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:03 smithi153 ceph-mon[25643]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:05 smithi153 ceph-mon[25643]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:06.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:05 smithi098 ceph-mon[24773]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:05 smithi098 ceph-mon[29473]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:07 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:07 smithi153 ceph-mon[25643]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:08.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:07 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:08.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:07 smithi098 ceph-mon[24773]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:07 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:07 smithi098 ceph-mon[29473]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:09 smithi153 ceph-mon[25643]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:09.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:41:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:41:09.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:09 smithi098 ceph-mon[24773]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:09.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:09 smithi098 ceph-mon[29473]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:09.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:41:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:09] "GET /metrics HTTP/1.1" 200 208008 "" "Prometheus/2.33.4" 2024-03-12T03:41:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:11 smithi153 ceph-mon[25643]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:12.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:11 smithi098 ceph-mon[24773]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:12.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:11 smithi098 ceph-mon[29473]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:13.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:13 smithi098 ceph-mon[24773]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:13.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:13 smithi098 ceph-mon[29473]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:13.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:41:13.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:13.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:13.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:41:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:13 smithi153 ceph-mon[25643]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:15 smithi153 ceph-mon[25643]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:16.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:15 smithi098 ceph-mon[24773]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:15 smithi098 ceph-mon[29473]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:17 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:17 smithi153 ceph-mon[25643]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:17 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:17 smithi098 ceph-mon[24773]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:17 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:17 smithi098 ceph-mon[29473]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:19 smithi153 ceph-mon[25643]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:19.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:41:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:41:19.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:19 smithi098 ceph-mon[24773]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:19.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:41:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:19] "GET /metrics HTTP/1.1" 200 208020 "" "Prometheus/2.33.4" 2024-03-12T03:41:19.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:19 smithi098 ceph-mon[29473]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:21 smithi153 ceph-mon[25643]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:21 smithi098 ceph-mon[24773]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:21 smithi098 ceph-mon[29473]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:23.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:23 smithi098 ceph-mon[24773]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:23.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:23 smithi098 ceph-mon[29473]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:23.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:41:23.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:41:23.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:41:23.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:23 smithi153 ceph-mon[25643]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:25 smithi153 ceph-mon[25643]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:26.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:25 smithi098 ceph-mon[24773]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:26.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:25 smithi098 ceph-mon[29473]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:27 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:27.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:27 smithi153 ceph-mon[25643]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:27 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:27 smithi098 ceph-mon[24773]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:27 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:27 smithi098 ceph-mon[29473]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:29.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:29 smithi153 ceph-mon[25643]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:29.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:41:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:41:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:29 smithi098 ceph-mon[24773]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:29 smithi098 ceph-mon[29473]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:29.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:41:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:29] "GET /metrics HTTP/1.1" 200 208014 "" "Prometheus/2.33.4" 2024-03-12T03:41:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:30 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:30 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:30.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:30 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:30.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:30 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:30.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:30 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:30.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:30 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:30.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:30 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:41:30.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:30 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:41:30.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:30 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:31.770 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:31 smithi153 ceph-mon[25643]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:31.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:31 smithi098 ceph-mon[24773]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:31.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:31 smithi098 ceph-mon[29473]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:33.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:33 smithi098 ceph-mon[24773]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:33 smithi098 ceph-mon[29473]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:41:33.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:33.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:41:33.899 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:33 smithi153 ceph-mon[25643]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:35.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:35 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:35.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:35 smithi098 ceph-mon[24773]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:35.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:35 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:35.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:35 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:35.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:35 smithi098 ceph-mon[29473]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:35.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:35 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:35 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:35.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:35 smithi153 ceph-mon[25643]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:35.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:35 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:41:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:37 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:37.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:37 smithi153 ceph-mon[25643]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:38.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:37 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:37 smithi098 ceph-mon[29473]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:37 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:37 smithi098 ceph-mon[24773]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:39.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:39 smithi153 ceph-mon[25643]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:39.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:41:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:41:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:39 smithi098 ceph-mon[24773]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:39 smithi098 ceph-mon[29473]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:39.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:41:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:39] "GET /metrics HTTP/1.1" 200 208014 "" "Prometheus/2.33.4" 2024-03-12T03:41:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:41:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:41:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:41:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:41:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:41:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:41 smithi153 ceph-mon[25643]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:42.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:41 smithi098 ceph-mon[24773]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:42.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:41 smithi098 ceph-mon[29473]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:43.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:43 smithi098 ceph-mon[24773]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:43.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:43 smithi098 ceph-mon[29473]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:43.826 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:41:43.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:43.826 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:43.826 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:43.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:41:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:43 smithi153 ceph-mon[25643]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:45.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:45 smithi153 ceph-mon[25643]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:46.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:45 smithi098 ceph-mon[24773]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:46.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:45 smithi098 ceph-mon[29473]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:47.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:47 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:47.911 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:47 smithi153 ceph-mon[25643]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:48.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:47 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:48.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:47 smithi098 ceph-mon[24773]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:47 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:47 smithi098 ceph-mon[29473]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:49 smithi153 ceph-mon[25643]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:49.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:41:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:41:49.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:49 smithi098 ceph-mon[24773]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:49.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:49 smithi098 ceph-mon[29473]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:49.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:41:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:49] "GET /metrics HTTP/1.1" 200 208012 "" "Prometheus/2.33.4" 2024-03-12T03:41:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:51 smithi153 ceph-mon[25643]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:52.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:51 smithi098 ceph-mon[24773]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:51 smithi098 ceph-mon[29473]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:53 smithi098 ceph-mon[24773]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:53 smithi098 ceph-mon[29473]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:41:53.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:53.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:41:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:41:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:41:53.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:41:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:53 smithi153 ceph-mon[25643]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:55 smithi153 ceph-mon[25643]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:56.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:55 smithi098 ceph-mon[24773]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:56.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:55 smithi098 ceph-mon[29473]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:57 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:57 smithi153 ceph-mon[25643]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:58.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:57 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:57 smithi098 ceph-mon[29473]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:57 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:41:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:57 smithi098 ceph-mon[24773]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:41:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:41:59 smithi153 ceph-mon[25643]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:59.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:41:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:41:59.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:41:59 smithi098 ceph-mon[24773]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:41:59.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:41:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:41:59] "GET /metrics HTTP/1.1" 200 208015 "" "Prometheus/2.33.4" 2024-03-12T03:41:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:41:59 smithi098 ceph-mon[29473]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:01 smithi153 ceph-mon[25643]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:02.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:01 smithi098 ceph-mon[24773]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:02.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:01 smithi098 ceph-mon[29473]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:03.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:03 smithi098 ceph-mon[24773]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:03 smithi098 ceph-mon[29473]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:03.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:42:03.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:03.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:03.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:03.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:03 smithi153 ceph-mon[25643]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:05 smithi153 ceph-mon[25643]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:06.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:05 smithi098 ceph-mon[24773]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:05 smithi098 ceph-mon[29473]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:07 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:07 smithi153 ceph-mon[25643]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:08.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:07 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:08.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:07 smithi098 ceph-mon[24773]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:07 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:07 smithi098 ceph-mon[29473]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:09 smithi153 ceph-mon[25643]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:09.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:42:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:42:09.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:09 smithi098 ceph-mon[24773]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:09 smithi098 ceph-mon[29473]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:09.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:42:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:09] "GET /metrics HTTP/1.1" 200 208015 "" "Prometheus/2.33.4" 2024-03-12T03:42:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:11 smithi153 ceph-mon[25643]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:12.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:11 smithi098 ceph-mon[24773]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:12.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:11 smithi098 ceph-mon[29473]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:13.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:13 smithi098 ceph-mon[24773]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:13.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:13 smithi098 ceph-mon[29473]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:13.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:42:13.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:13.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:13.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:13.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:13.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:13 smithi153 ceph-mon[25643]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:15 smithi153 ceph-mon[25643]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:16.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:15 smithi098 ceph-mon[24773]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:15 smithi098 ceph-mon[29473]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:17 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:17.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:17 smithi153 ceph-mon[25643]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:18.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:17 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:17 smithi098 ceph-mon[29473]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:17 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:17 smithi098 ceph-mon[24773]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:19 smithi153 ceph-mon[25643]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:19.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:42:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:19] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:42:19.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:19 smithi098 ceph-mon[24773]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:19.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:19 smithi098 ceph-mon[29473]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:19.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:42:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:19] "GET /metrics HTTP/1.1" 200 208021 "" "Prometheus/2.33.4" 2024-03-12T03:42:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:21 smithi153 ceph-mon[25643]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:21 smithi098 ceph-mon[24773]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:21 smithi098 ceph-mon[29473]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:23.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:23 smithi098 ceph-mon[24773]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:23.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:23 smithi098 ceph-mon[29473]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:23.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:42:23.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:23.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:23.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:23.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:23.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:23 smithi153 ceph-mon[25643]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:25 smithi153 ceph-mon[25643]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:26.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:25 smithi098 ceph-mon[24773]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:26.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:25 smithi098 ceph-mon[29473]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:27 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:27 smithi153 ceph-mon[25643]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:28.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:27 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:27 smithi098 ceph-mon[24773]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:27 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:27 smithi098 ceph-mon[29473]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:29 smithi153 ceph-mon[25643]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:29.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:42:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:42:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:29 smithi098 ceph-mon[24773]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:29 smithi098 ceph-mon[29473]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:29.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:42:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:29] "GET /metrics HTTP/1.1" 200 208007 "" "Prometheus/2.33.4" 2024-03-12T03:42:31.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:31 smithi153 ceph-mon[25643]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:32.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:31 smithi098 ceph-mon[24773]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:32.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:31 smithi098 ceph-mon[29473]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:33.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:33 smithi098 ceph-mon[24773]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:33 smithi098 ceph-mon[29473]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:42:33.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:33.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:33.825 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:33.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:33 smithi153 ceph-mon[25643]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:34.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:34 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:34.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:34 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:34.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:34 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:34.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:34 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:34.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:34 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:42:34.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:34 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:42:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:35 smithi153 ceph-mon[25643]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:35.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:35 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:42:36.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:35 smithi098 ceph-mon[24773]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:36.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:35 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:42:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:35 smithi098 ceph-mon[29473]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:35 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' 2024-03-12T03:42:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:37 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:37.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:37 smithi153 ceph-mon[25643]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:38.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:37 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:38.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:37 smithi098 ceph-mon[24773]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:37 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:38.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:37 smithi098 ceph-mon[29473]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:39 smithi153 ceph-mon[25643]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:39.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:42:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:42:39.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:39 smithi098 ceph-mon[24773]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:39 smithi098 ceph-mon[29473]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:39.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:42:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:39] "GET /metrics HTTP/1.1" 200 208007 "" "Prometheus/2.33.4" 2024-03-12T03:42:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:42:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:42:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:40 smithi153 ceph-mon[25643]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:42:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:40 smithi153 ceph-mon[25643]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[24773]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[24773]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[29473]: from='mgr.24343 172.21.15.98:0/3497550317' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:42:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:40 smithi098 ceph-mon[29473]: from='mgr.24343 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:42:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:41 smithi153 ceph-mon[25643]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:42.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:41 smithi098 ceph-mon[24773]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:42.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:41 smithi098 ceph-mon[29473]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:43.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:43 smithi098 ceph-mon[24773]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:43 smithi098 ceph-mon[29473]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:43.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:42:43.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:43.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:43.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:43.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:43.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:43 smithi153 ceph-mon[25643]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:45.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:45 smithi153 ceph-mon[25643]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:46.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:45 smithi098 ceph-mon[24773]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:46.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:45 smithi098 ceph-mon[29473]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:47 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:47.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:47 smithi153 ceph-mon[25643]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:48.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:47 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:48.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:47 smithi098 ceph-mon[24773]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:47 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:47 smithi098 ceph-mon[29473]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:49 smithi153 ceph-mon[25643]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:49.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:42:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:42:49.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:49 smithi098 ceph-mon[24773]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:49.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:49 smithi098 ceph-mon[29473]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:49.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:42:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:49] "GET /metrics HTTP/1.1" 200 208013 "" "Prometheus/2.33.4" 2024-03-12T03:42:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:51 smithi153 ceph-mon[25643]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:52.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:51 smithi098 ceph-mon[24773]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:51 smithi098 ceph-mon[29473]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:53.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:53 smithi098 ceph-mon[24773]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:53 smithi098 ceph-mon[29473]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:42:53.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:42:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:42:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:42:53.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:42:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:53 smithi153 ceph-mon[25643]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:55 smithi153 ceph-mon[25643]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:56.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:55 smithi098 ceph-mon[24773]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:56.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:55 smithi098 ceph-mon[29473]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:57 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:57 smithi153 ceph-mon[25643]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:57 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:58.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:57 smithi098 ceph-mon[24773]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:57 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:42:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:57 smithi098 ceph-mon[29473]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:42:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:42:59 smithi153 ceph-mon[25643]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:59.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:42:59 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:59] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:42:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:42:59 smithi098 ceph-mon[24773]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:42:59.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:42:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:42:59] "GET /metrics HTTP/1.1" 200 208013 "" "Prometheus/2.33.4" 2024-03-12T03:42:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:42:59 smithi098 ceph-mon[29473]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:01 smithi153 ceph-mon[25643]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:02.049 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:01 smithi098 ceph-mon[24773]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:02.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:01 smithi098 ceph-mon[29473]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:02.819 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T03:43:03.656 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:43:03.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:43:03.656 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:43:03.656 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:43:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:03 smithi153 ceph-mon[25643]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:04.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:03 smithi098 ceph-mon[24773]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:04.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:03 smithi098 ceph-mon[29473]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (6m) 2m ago 7m 25.5M - ba2b418f427c c12f6c606e5f 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (6m) 90s ago 6m 56.0M - 8.3.5 dad864ee21e9 7864a9bab9e4 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (6m) 2m ago 6m 51.0M - 3.5 e1d6a67b021e 4c7a21937fd9 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283 running (2m) 90s ago 11m 449M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 2e7f2996e113 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:9283 running (13m) 2m ago 13m 460M - 17.2.0 e1d6a67b021e 3c66c3a9135b 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (13m) 2m ago 13m 49.5M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (11m) 90s ago 11m 41.1M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (12m) 2m ago 12m 41.4M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:43:04.747 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (8m) 2m ago 8m 19.4M - 1dbe0e931976 fd51a335b42c 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (7m) 90s ago 7m 17.5M - 1dbe0e931976 270f07cc8db8 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (11m) 2m ago 11m 51.8M 4016M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (11m) 2m ago 11m 53.2M 4016M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (10m) 2m ago 10m 46.9M 4016M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (10m) 2m ago 10m 49.2M 4016M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (9m) 90s ago 9m 53.0M 4272M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (9m) 90s ago 9m 50.9M 4272M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (9m) 90s ago 9m 47.7M 4272M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (8m) 90s ago 8m 49.6M 4272M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (6m) 90s ago 8m 68.8M - 514e6a882f6e 7edcfd1725dd 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (6m) 2m ago 6m 70.7M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:43:04.748 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (6m) 90s ago 6m 74.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:43:05.330 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions' 2024-03-12T03:43:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:05 smithi153 ceph-mon[25643]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:05.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:05 smithi098 ceph-mon[24773]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:05.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:05 smithi098 ceph-mon[29473]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:06 smithi098 ceph-mon[24773]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:43:06.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:06 smithi098 ceph-mon[29473]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:43:06.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:06 smithi153 ceph-mon[25643]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 1 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:43:07.223 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 1 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:43:07.224 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:43:07.843 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph -s' 2024-03-12T03:43:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:07 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:07.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:07 smithi153 ceph-mon[25643]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:07.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:07 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2539049230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:43:08.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:07 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:08.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:07 smithi098 ceph-mon[24773]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:08.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:07 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2539049230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:43:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:07 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:07 smithi098 ceph-mon[29473]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:07 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2539049230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:43:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:09 smithi153 ceph-mon[25643]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:09.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:09 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:43:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:09 smithi098 ceph-mon[24773]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:09 smithi098 ceph-mon[29473]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:09.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:09] "GET /metrics HTTP/1.1" 200 208013 "" "Prometheus/2.33.4" 2024-03-12T03:43:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: cluster: 2024-03-12T03:43:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: id: ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:43:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: health: HEALTH_OK 2024-03-12T03:43:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:43:09.853 INFO:teuthology.orchestra.run.smithi098.stdout: services: 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: mon: 3 daemons, quorum a,c,b (age 11m) 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: mgr: y(active, since 8m), standbys: x 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 9m) 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: data: 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: pools: 6 pools, 161 pgs 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: objects: 241 objects, 585 KiB 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: usage: 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: pgs: 161 active+clean 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: io: 2024-03-12T03:43:09.854 INFO:teuthology.orchestra.run.smithi098.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:43:09.855 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:43:10.440 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T03:43:10.747 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:10 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1198848421' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:43:10.747 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:10 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1198848421' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:43:10.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:10 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1198848421' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:43:11.698 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:11 smithi098 ceph-mon[24773]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:11.699 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:11 smithi098 ceph-mon[29473]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:11 smithi153 ceph-mon[25643]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:12.433 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T03:43:12.756 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:12 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/332477133' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:43:12.756 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:12 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/332477133' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:43:12.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:12 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/332477133' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:43:13.003 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-03-12T03:43:13.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:13 smithi098 ceph-mon[24773]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:13.559 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:13 smithi098 ceph-mon[29473]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:13.559 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:43:13.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:43:13.559 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:13.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:43:13.559 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:13.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:43:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:13 smithi153 ceph-mon[25643]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:14.909 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:43:15.485 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph mgr fail' 2024-03-12T03:43:15.910 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:15 smithi098 ceph-mon[24773]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:15.910 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:15 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3922765903' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:43:15.911 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:15 smithi098 ceph-mon[29473]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:15.911 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:15 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3922765903' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:43:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:15 smithi153 ceph-mon[25643]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:15 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3922765903' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-mon[25643]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/576938483' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-mon[25643]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:43:17] ENGINE Bus STOPPING 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:43:17] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:43:17] ENGINE Bus STOPPED 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:43:17] ENGINE Bus STARTING 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:43:17] ENGINE Serving on http://:::9283 2024-03-12T03:43:17.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:43:17] ENGINE Bus STARTED 2024-03-12T03:43:17.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:17.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[24773]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:17.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/576938483' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:43:17.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:43:17.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[24773]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:43:17.946 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:17.946 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[29473]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:17.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/576938483' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:43:17.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:43:17.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mon[29473]: osdmap e96: 8 total, 8 up, 8 in 2024-03-12T03:43:17.947 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:17.660+0000 7fc9f9eee700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-12T03:43:17.947 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setuser ceph since I am not root 2024-03-12T03:43:17.947 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ignoring --setgroup ceph since I am not root 2024-03-12T03:43:17.947 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-mgr[25009]: -- 172.21.15.98:0/2208334118 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563bd117c340 con 0x563bd1154c00 2024-03-12T03:43:18.278 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:17 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:17.979Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T03:43:18.278 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:18.180+0000 7f29d42e0000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:43:18.278 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:18.275+0000 7f29d42e0000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:43:18.351 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'sleep 180' 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: mgrmap e22: x(active, starting, since 0.212529s) 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:43:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: Manager daemon x is now available 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: Queued rgw.foo for migration 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:43:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:43:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:43:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:43:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:18 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.967 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:18.906+0000 7f29d42e0000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:43:18.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:43:18.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: mgrmap e22: x(active, starting, since 0.212529s) 2024-03-12T03:43:18.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:43:18.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:43:18.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: Manager daemon x is now available 2024-03-12T03:43:18.968 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: Queued rgw.foo for migration 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: mgrmap e22: x(active, starting, since 0.212529s) 2024-03-12T03:43:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:43:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: Manager daemon x is now available 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: Queued rgw.foo for migration 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:43:18.971 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:43:18.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:43:18.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.972 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:43:18.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:18.972 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:19.236 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:18.965Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T03:43:19.236 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.000+0000 7f29d42e0000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:43:19.236 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.083+0000 7f29d42e0000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:43:19.514 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.233+0000 7f29d42e0000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:43:19.514 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.317+0000 7f29d42e0000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:43:19.514 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.427+0000 7f29d42e0000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:43:19.514 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.511+0000 7f29d42e0000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:43:19.679 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:19] "GET /metrics HTTP/1.1" 200 34896 "" "Prometheus/2.33.4" 2024-03-12T03:43:19.765 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.688+0000 7f29d42e0000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:43:19.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[24773]: Deploying cephadm binary to smithi098 2024-03-12T03:43:19.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[24773]: Deploying cephadm binary to smithi153 2024-03-12T03:43:19.765 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[24773]: mgrmap e23: x(active, since 1.22743s) 2024-03-12T03:43:19.766 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[24773]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:19.766 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[24773]: [12/Mar/2024:03:43:18] ENGINE Bus STARTING 2024-03-12T03:43:19.766 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[29473]: Deploying cephadm binary to smithi098 2024-03-12T03:43:19.766 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[29473]: Deploying cephadm binary to smithi153 2024-03-12T03:43:19.766 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[29473]: mgrmap e23: x(active, since 1.22743s) 2024-03-12T03:43:19.766 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[29473]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:19.766 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-mon[29473]: [12/Mar/2024:03:43:18] ENGINE Bus STARTING 2024-03-12T03:43:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:19 smithi153 ceph-mon[25643]: Deploying cephadm binary to smithi098 2024-03-12T03:43:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:19 smithi153 ceph-mon[25643]: Deploying cephadm binary to smithi153 2024-03-12T03:43:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:19 smithi153 ceph-mon[25643]: mgrmap e23: x(active, since 1.22743s) 2024-03-12T03:43:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:19 smithi153 ceph-mon[25643]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:19 smithi153 ceph-mon[25643]: [12/Mar/2024:03:43:18] ENGINE Bus STARTING 2024-03-12T03:43:20.059 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:19 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:19.776+0000 7f29d42e0000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:43:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:20 smithi153 ceph-mon[25643]: [12/Mar/2024:03:43:19] ENGINE Serving on https://172.21.15.153:7150 2024-03-12T03:43:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:20 smithi153 ceph-mon[25643]: [12/Mar/2024:03:43:19] ENGINE Serving on http://172.21.15.153:8765 2024-03-12T03:43:20.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:20 smithi153 ceph-mon[25643]: [12/Mar/2024:03:43:19] ENGINE Bus STARTED 2024-03-12T03:43:20.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:20 smithi153 ceph-mon[25643]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:21.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[24773]: [12/Mar/2024:03:43:19] ENGINE Serving on https://172.21.15.153:7150 2024-03-12T03:43:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[24773]: [12/Mar/2024:03:43:19] ENGINE Serving on http://172.21.15.153:8765 2024-03-12T03:43:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[24773]: [12/Mar/2024:03:43:19] ENGINE Bus STARTED 2024-03-12T03:43:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[24773]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:21.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:20.799+0000 7f29d42e0000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:43:21.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:20.927+0000 7f29d42e0000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:43:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[29473]: [12/Mar/2024:03:43:19] ENGINE Serving on https://172.21.15.153:7150 2024-03-12T03:43:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[29473]: [12/Mar/2024:03:43:19] ENGINE Serving on http://172.21.15.153:8765 2024-03-12T03:43:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[29473]: [12/Mar/2024:03:43:19] ENGINE Bus STARTED 2024-03-12T03:43:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:20 smithi098 ceph-mon[29473]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:21.521 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:21.116+0000 7f29d42e0000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:43:21.521 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:21.204+0000 7f29d42e0000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:43:21.522 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:21.339+0000 7f29d42e0000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:43:21.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-mon[24773]: mgrmap e24: x(active, since 3s) 2024-03-12T03:43:21.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-mon[29473]: mgrmap e24: x(active, since 3s) 2024-03-12T03:43:21.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:21.661+0000 7f29d42e0000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:43:21.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:21.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:43:22.140 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:21.884+0000 7f29d42e0000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:43:22.140 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:21.969+0000 7f29d42e0000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:43:22.140 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:22 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:22.137+0000 7f29d42e0000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:43:22.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:21 smithi153 ceph-mon[25643]: mgrmap e24: x(active, since 3s) 2024-03-12T03:43:23.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:22 smithi098 ceph-mon[24773]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:23.068 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:22 smithi098 ceph-mon[29473]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:22 smithi153 ceph-mon[25643]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:23.326 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:23.065+0000 7f29d42e0000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:43:23.651 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:23.382+0000 7f29d42e0000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:43:23.651 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:23.487+0000 7f29d42e0000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:43:23.651 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:23.647+0000 7f29d42e0000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:43:23.651 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:43:23.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T03:43:23.651 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T03:43:23.651 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:23.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:43:24.055 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:43:23] ENGINE Bus STARTING 2024-03-12T03:43:24.055 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: CherryPy Checker: 2024-03-12T03:43:24.055 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: The Application mounted at '' has an empty config. 2024-03-12T03:43:24.055 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: 2024-03-12T03:43:24.055 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:43:23] ENGINE Serving on http://:::9283 2024-03-12T03:43:24.055 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: [12/Mar/2024:03:43:23] ENGINE Bus STARTED 2024-03-12T03:43:24.323 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:24.053Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:43:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:24 smithi153 ceph-mon[25643]: Standby manager daemon y started 2024-03-12T03:43:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:24 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:43:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:24 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:43:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:24 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:43:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:24 smithi153 ceph-mon[25643]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:24 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:43:24.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[29473]: Standby manager daemon y started 2024-03-12T03:43:24.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:43:24.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:43:24.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:43:24.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[29473]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:24.818 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:43:24.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[24773]: Standby manager daemon y started 2024-03-12T03:43:24.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:43:24.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:43:24.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:43:24.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[24773]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:43:24.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:24 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/2715202831' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:43:25.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:25 smithi098 ceph-mon[24773]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-12T03:43:25.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:25 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:43:25.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:25 smithi098 ceph-mon[29473]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-12T03:43:25.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:25 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:43:25.828 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:25 smithi153 ceph-mon[25643]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-12T03:43:25.829 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:25 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:43:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:26 smithi153 ceph-mon[25643]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:43:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:26 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:26.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:26 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:26.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:26 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:26.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[29473]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:43:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[24773]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:43:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:26 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:27 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:28.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:27 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi098 to 2992M 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: Adjusting osd_memory_target on smithi153 to 3248M 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi098 to 2992M 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: Adjusting osd_memory_target on smithi153 to 3248M 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.327 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi098 to 2992M 2024-03-12T03:43:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:43:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.387 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: Adjusting osd_memory_target on smithi153 to 3248M 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.388 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.389 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:29.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:29] "GET /metrics HTTP/1.1" 200 34896 "" "Prometheus/2.33.4" 2024-03-12T03:43:29.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:29] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:43:29.824 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:29 smithi098 systemd[1]: Stopping Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:29.824 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:29 smithi098 bash[58486]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter.a" found: no such container 2024-03-12T03:43:29.824 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:29 smithi098 podman[58499]: 2024-03-12 03:43:29.698715044 +0000 UTC m=+0.138244100 container died fd51a335b42c7cc1f8b575ea8ce74da6b45af3e997d091cf2d6ee7b06ab7ce90 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:30.324 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 podman[58499]: 2024-03-12 03:43:30.10491203 +0000 UTC m=+0.544441097 container cleanup fd51a335b42c7cc1f8b575ea8ce74da6b45af3e997d091cf2d6ee7b06ab7ce90 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:30.324 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 bash[58499]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a 2024-03-12T03:43:30.324 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 bash[58526]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter.a" found: no such container 2024-03-12T03:43:30.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:43:30.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:43:30.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:43:30.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:43:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:29 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:29 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:30.703 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 podman[58514]: 2024-03-12 03:43:30.348319714 +0000 UTC m=+0.653494321 container remove fd51a335b42c7cc1f8b575ea8ce74da6b45af3e997d091cf2d6ee7b06ab7ce90 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:30.703 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:43:30.703 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-12T03:43:30.703 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 systemd[1]: Stopped Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:30.703 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 3.675s CPU time. 2024-03-12T03:43:30.704 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 systemd[1]: Starting Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:30.704 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 podman[58601]: 2024-03-12 03:43:30.481983616 +0000 UTC m=+0.017539848 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-12T03:43:30.704 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 podman[58601]: 2024-03-12 03:43:30.590439197 +0000 UTC m=+0.125995417 container create ff35fa2e462e9753a01df77fc95dd029cc4af22dfdcda708304730f7e9bd4c33 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:31.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[24773]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-12T03:43:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[24773]: Reconfiguring daemon node-exporter.a on smithi098 2024-03-12T03:43:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[24773]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:43:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[29473]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-12T03:43:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[29473]: Reconfiguring daemon node-exporter.a on smithi098 2024-03-12T03:43:31.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[29473]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:43:31.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:31.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:31.075 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 podman[58601]: 2024-03-12 03:43:30.809924041 +0000 UTC m=+0.345480277 container init ff35fa2e462e9753a01df77fc95dd029cc4af22dfdcda708304730f7e9bd4c33 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:31.075 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 podman[58601]: 2024-03-12 03:43:30.812636195 +0000 UTC m=+0.348192414 container start ff35fa2e462e9753a01df77fc95dd029cc4af22dfdcda708304730f7e9bd4c33 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:31.075 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.815Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-12T03:43:31.075 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.815Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.816Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.816Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.816Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=arp 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-12T03:43:31.076 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=edac 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-12T03:43:31.077 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=os 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=stat 2024-03-12T03:43:31.078 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=time 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=uname 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[58615]: ts=2024-03-12T03:43:30.817Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 bash[58601]: ff35fa2e462e9753a01df77fc95dd029cc4af22dfdcda708304730f7e9bd4c33 2024-03-12T03:43:31.079 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:43:30 smithi098 systemd[1]: Started Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:31.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:30 smithi153 ceph-mon[25643]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-12T03:43:31.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:30 smithi153 ceph-mon[25643]: Reconfiguring daemon node-exporter.a on smithi098 2024-03-12T03:43:31.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:30 smithi153 ceph-mon[25643]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:43:31.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:30 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:31.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:30 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:32.315 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:31 smithi153 ceph-mon[25643]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:43:32.315 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:31 smithi153 ceph-mon[25643]: Reconfiguring daemon alertmanager.a on smithi098 2024-03-12T03:43:32.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:31 smithi098 ceph-mon[24773]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:43:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:31 smithi098 ceph-mon[24773]: Reconfiguring daemon alertmanager.a on smithi098 2024-03-12T03:43:32.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:31 smithi098 ceph-mon[29473]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-12T03:43:32.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:31 smithi098 ceph-mon[29473]: Reconfiguring daemon alertmanager.a on smithi098 2024-03-12T03:43:32.692 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:32.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.98:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.153:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:43:33.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:32 smithi098 ceph-mon[24773]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:33.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:32 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:33.236 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:32 smithi098 ceph-mon[29473]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:33.237 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:32 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:32 smithi153 ceph-mon[25643]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:32 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=error ts=2024-03-12T03:43:33.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:43:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:33.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.98:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.98 because it doesn't contain any IP SANs" 2024-03-12T03:43:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=warn ts=2024-03-12T03:43:33.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.153:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.153 because it doesn't contain any IP SANs" 2024-03-12T03:43:34.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:34 smithi098 ceph-mon[24773]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:34.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:34 smithi098 ceph-mon[29473]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:34 smithi153 ceph-mon[25643]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:36.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:36 smithi098 ceph-mon[29473]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:36.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:36 smithi098 ceph-mon[29473]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:36.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:36 smithi098 ceph-mon[24773]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:36.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:36 smithi098 ceph-mon[24773]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:36.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:36 smithi153 ceph-mon[25643]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:43:36.990 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:36 smithi153 ceph-mon[25643]: from='client.34613 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:43:38.074 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:37 smithi098 systemd[1]: Stopping Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:38.074 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:37 smithi098 bash[58883]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-12T03:43:38.074 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[49636]: level=info ts=2024-03-12T03:43:37.841Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:43:38.074 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:37 smithi098 podman[58895]: 2024-03-12 03:43:37.862285999 +0000 UTC m=+0.148293081 container died c12f6c606e5fc4254907ddc9fe3a4ca0a32c2ed9a2b776635ad8075ff2d3aa0c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:38.511 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 podman[58895]: 2024-03-12 03:43:38.25006549 +0000 UTC m=+0.536072566 container cleanup c12f6c606e5fc4254907ddc9fe3a4ca0a32c2ed9a2b776635ad8075ff2d3aa0c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:38.511 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 bash[58895]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:43:38.511 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 bash[58921]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager.a" found: no such container 2024-03-12T03:43:38.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:38 smithi098 ceph-mon[24773]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:38.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:38 smithi098 ceph-mon[29473]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:38.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 podman[58909]: 2024-03-12 03:43:38.509503781 +0000 UTC m=+0.642829971 container remove c12f6c606e5fc4254907ddc9fe3a4ca0a32c2ed9a2b776635ad8075ff2d3aa0c (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:38.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 podman[58909]: 2024-03-12 03:43:38.610435806 +0000 UTC m=+0.743761952 volume remove cddf350d208d22949551d0f9fa665be999d7e04ad99aa40bf98eaa8cf1cb49d1 2024-03-12T03:43:38.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:43:38.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 systemd[1]: Stopped Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:38.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.063s CPU time. 2024-03-12T03:43:38.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 systemd[1]: Starting Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:39.126 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 podman[58983]: 2024-03-12 03:43:38.748166152 +0000 UTC m=+0.018481019 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-12T03:43:39.126 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 podman[58983]: 2024-03-12 03:43:38.846133837 +0000 UTC m=+0.116448748 volume create 43f5f9a50572ba1c177c6dcb6001e8987e29de5da7d0f758783ae27bd531ed12 2024-03-12T03:43:39.126 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:38 smithi098 podman[58983]: 2024-03-12 03:43:38.978306939 +0000 UTC m=+0.248621805 container create ae93804061060426dd9dc3bb7a467194cbdfbf6744047ea7e6b81c507955acc7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:39.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:38 smithi153 ceph-mon[25643]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:39.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 podman[58983]: 2024-03-12 03:43:39.248054472 +0000 UTC m=+0.518369338 container init ae93804061060426dd9dc3bb7a467194cbdfbf6744047ea7e6b81c507955acc7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:39.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 podman[58983]: 2024-03-12 03:43:39.251088341 +0000 UTC m=+0.521403210 container start ae93804061060426dd9dc3bb7a467194cbdfbf6744047ea7e6b81c507955acc7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:39.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.261Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-12T03:43:39.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.261Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-12T03:43:39.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.263Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.98 port=9094 2024-03-12T03:43:39.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.264Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:43:39.575 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.289Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:43:39.575 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.290Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:43:39.575 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.292Z caller=main.go:518 msg=Listening address=:9093 2024-03-12T03:43:39.575 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:39.292Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-12T03:43:39.575 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 bash[58983]: ae93804061060426dd9dc3bb7a467194cbdfbf6744047ea7e6b81c507955acc7 2024-03-12T03:43:39.575 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:39 smithi098 systemd[1]: Started Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:39.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:39] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-03-12T03:43:39.986 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:43:40.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[24773]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:40.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:40.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:43:40.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:43:40.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:43:40.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:40.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:43:40.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:40 smithi098 ceph-mon[29473]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:43:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:43:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:43:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:43:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:43:40.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:40 smithi153 ceph-mon[25643]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:43:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:41.265Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00092246s 2024-03-12T03:43:42.692 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:42.313Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.98:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.153:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:43:42.392+0000 7f056598d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Creating ceph-iscsi config... 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Traceback (most recent call last): 2024-03-12T03:43:42.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return _run_code(code, main_globals, None, 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: exec(code, run_globals) 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:42.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Traceback (most recent call last): 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:43:42.695 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return future.result(timeout) 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return self.__get_result() 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: raise self._exception 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: config_blobs=daemon_spec.final_config, 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Creating ceph-iscsi config... 2024-03-12T03:43:42.696 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Traceback (most recent call last): 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return _run_code(code, main_globals, None, 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: exec(code, run_globals) 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:42.697 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:42.698 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:42.698 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:42.698 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:42.698 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:43.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:42 smithi098 ceph-mon[24773]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:42 smithi098 ceph-mon[29473]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:42 smithi153 ceph-mon[25643]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:43 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:43:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:44.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:43:44.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:44.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:43 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:44.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:43:44.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:44.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:44.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:43 smithi098 ceph-mon[29473]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:43 smithi098 ceph-mon[29473]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:43 smithi098 ceph-mon[24773]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:43:44.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:43 smithi098 ceph-mon[24773]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:43 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:43:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:43:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:43:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:43 smithi153 ceph-mon[25643]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:43:44.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:43 smithi153 ceph-mon[25643]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:43:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:44 smithi098 ceph-mon[24773]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:44 smithi098 ceph-mon[29473]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:44 smithi153 ceph-mon[25643]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:43:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:46 smithi098 ceph-mon[24773]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:43:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:46 smithi098 ceph-mon[29473]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:43:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:46 smithi153 ceph-mon[25643]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:43:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:48 smithi098 ceph-mon[24773]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:43:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:48 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:48 smithi098 ceph-mon[29473]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:43:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:48 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:48 smithi153 ceph-mon[25643]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:43:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:48 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:43:49.573 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:43:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:43:49.268Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003860038s 2024-03-12T03:43:49.633 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:43:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:49] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-03-12T03:43:50.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:43:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[25005]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:43:49] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:43:51.020 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:50 smithi153 ceph-mon[25643]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:43:51.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:50 smithi098 ceph-mon[24773]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:43:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:50 smithi098 ceph-mon[29473]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:43:52.035 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:51 smithi153 systemd[1]: Stopping Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:52.036 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:51 smithi153 bash[49022]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.033Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.033Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-12T03:43:52.320 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[42567]: ts=2024-03-12T03:43:52.034Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-12T03:43:52.321 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 podman[49033]: 2024-03-12 03:43:52.044156995 +0000 UTC m=+0.127433622 container died 7edcfd1725dd9b292c4bbb9c2411ba7ff3a9ed29d19aa5ee0c7118fd3afbb9e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:52.678 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 podman[49033]: 2024-03-12 03:43:52.433718061 +0000 UTC m=+0.516994690 container cleanup 7edcfd1725dd9b292c4bbb9c2411ba7ff3a9ed29d19aa5ee0c7118fd3afbb9e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:52.678 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 bash[49033]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:43:52.678 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 bash[49058]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:43:52.937 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:52 smithi153 ceph-mon[25643]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:52.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 podman[49047]: 2024-03-12 03:43:52.676713302 +0000 UTC m=+0.630722895 container remove 7edcfd1725dd9b292c4bbb9c2411ba7ff3a9ed29d19aa5ee0c7118fd3afbb9e7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:52.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:43:52.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 systemd[1]: Stopped Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:52.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.958s CPU time. 2024-03-12T03:43:52.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 systemd[1]: Starting Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:52.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 podman[49123]: 2024-03-12 03:43:52.806946811 +0000 UTC m=+0.018730362 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-12T03:43:52.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:52 smithi153 podman[49123]: 2024-03-12 03:43:52.936263717 +0000 UTC m=+0.148047264 container create 8b54432f20549a03a39fc8d1c7251e9692f81e098716fe68c780db8bf65270f0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:53.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:52 smithi098 ceph-mon[24773]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:53.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:52 smithi098 ceph-mon[29473]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:53.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 podman[49123]: 2024-03-12 03:43:53.121721101 +0000 UTC m=+0.333504651 container init 8b54432f20549a03a39fc8d1c7251e9692f81e098716fe68c780db8bf65270f0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:53.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 podman[49123]: 2024-03-12 03:43:53.12437312 +0000 UTC m=+0.336156667 container start 8b54432f20549a03a39fc8d1c7251e9692f81e098716fe68c780db8bf65270f0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:43:53.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.148Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-12T03:43:53.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.148Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-12T03:43:53.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.148Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-12T03:43:53.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.148Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi153 (none))" 2024-03-12T03:43:53.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.148Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:43:53.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.148Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:43:53.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.150Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:43:53.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.150Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-12T03:43:53.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.151Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-12T03:43:53.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.153Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:43:53.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.153Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.84µs 2024-03-12T03:43:53.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.153Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:43:53.608 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 bash[49123]: 8b54432f20549a03a39fc8d1c7251e9692f81e098716fe68c780db8bf65270f0 2024-03-12T03:43:53.608 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 systemd[1]: Started Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:53.608 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.358Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-12T03:43:53.608 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.606Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-12T03:43:53.943 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:53 smithi153 systemd[1]: Stopping Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:53.943 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:53 smithi153 bash[49246]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter.b" found: no such container 2024-03-12T03:43:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.606Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-12T03:43:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.606Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=29.739µs wal_replay_duration=452.647531ms total_replay_duration=452.690906ms 2024-03-12T03:43:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.609Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:43:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.609Z caller=main.go:947 level=info msg="TSDB started" 2024-03-12T03:43:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.609Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:43:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.633Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.108467ms db_storage=683ns remote_storage=1.656µs web_handler=398ns query_engine=593ns scrape=354.369µs scrape_sd=104.371µs notify=11.249µs notify_sd=15.863µs rules=23.314242ms 2024-03-12T03:43:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:43:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:43:53.633Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-12T03:43:54.253 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:53 smithi153 podman[49257]: 2024-03-12 03:43:53.977291475 +0000 UTC m=+0.135696742 container died 270f07cc8db829f7a928d4a750cd395bf4274a986e0452426ce472939643528c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: Reconfiguring daemon node-exporter.b on smithi153 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2498702603' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/852300963' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]: dispatch 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]: dispatch 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:54.540 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:54.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-12T03:43:54.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: Reconfiguring daemon node-exporter.b on smithi153 2024-03-12T03:43:54.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:54.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2498702603' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:43:54.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/852300963' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]: dispatch 2024-03-12T03:43:54.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:54 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]: dispatch 2024-03-12T03:43:54.588 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 podman[49257]: 2024-03-12 03:43:54.35007007 +0000 UTC m=+0.508475332 container cleanup 270f07cc8db829f7a928d4a750cd395bf4274a986e0452426ce472939643528c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:43:54.588 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 bash[49257]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b 2024-03-12T03:43:54.588 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 bash[49285]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter.b" found: no such container 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: Reconfiguring daemon node-exporter.b on smithi153 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2498702603' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/852300963' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]: dispatch 2024-03-12T03:43:54.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:54 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]: dispatch 2024-03-12T03:43:54.854 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 podman[49272]: 2024-03-12 03:43:54.622527685 +0000 UTC m=+0.649943257 container remove 270f07cc8db829f7a928d4a750cd395bf4274a986e0452426ce472939643528c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:43:54.854 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:43:54.854 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-12T03:43:54.854 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 systemd[1]: Stopped Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:54.855 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 3.741s CPU time. 2024-03-12T03:43:54.855 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 systemd[1]: Starting Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:43:55.131 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 podman[49361]: 2024-03-12 03:43:54.852239827 +0000 UTC m=+0.111942855 container create 78fb1d60e45411315445059920e95429426d7be9601168415e1bc80470c6953c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:54 smithi153 podman[49361]: 2024-03-12 03:43:54.759123824 +0000 UTC m=+0.018826856 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 podman[49361]: 2024-03-12 03:43:55.054519453 +0000 UTC m=+0.314222473 container init 78fb1d60e45411315445059920e95429426d7be9601168415e1bc80470c6953c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 podman[49361]: 2024-03-12 03:43:55.057048341 +0000 UTC m=+0.316751371 container start 78fb1d60e45411315445059920e95429426d7be9601168415e1bc80470c6953c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.060Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.060Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=arp 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-12T03:43:55.132 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=edac 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-12T03:43:55.133 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=os 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-12T03:43:55.134 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=stat 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=time 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=uname 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.061Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[49378]: ts=2024-03-12T03:43:55.062Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-12T03:43:55.135 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 bash[49361]: 78fb1d60e45411315445059920e95429426d7be9601168415e1bc80470c6953c 2024-03-12T03:43:55.442 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:43:55 smithi153 systemd[1]: Started Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:43:55.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]': finished 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1860640337' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]: dispatch 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]: dispatch 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]': finished 2024-03-12T03:43:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:43:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1860640337' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]: dispatch 2024-03-12T03:43:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]: dispatch 2024-03-12T03:43:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:55 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:43:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2704764386"}]': finished 2024-03-12T03:43:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: osdmap e97: 8 total, 8 up, 8 in 2024-03-12T03:43:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1860640337' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]: dispatch 2024-03-12T03:43:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]: dispatch 2024-03-12T03:43:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:43:55.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:55 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: Reconfiguring grafana.a (dependencies changed)... 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: Reconfiguring daemon grafana.a on smithi153 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]': finished 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2623870350' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3833154681"}]: dispatch 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: Reconfiguring grafana.a (dependencies changed)... 2024-03-12T03:43:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:43:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:43:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: Reconfiguring daemon grafana.a on smithi153 2024-03-12T03:43:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]': finished 2024-03-12T03:43:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:43:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-12T03:43:56.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:56 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2623870350' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3833154681"}]: dispatch 2024-03-12T03:43:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: Reconfiguring grafana.a (dependencies changed)... 2024-03-12T03:43:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-12T03:43:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-12T03:43:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: Reconfiguring daemon grafana.a on smithi153 2024-03-12T03:43:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1273287586"}]': finished 2024-03-12T03:43:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: osdmap e98: 8 total, 8 up, 8 in 2024-03-12T03:43:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s 2024-03-12T03:43:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:56 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2623870350' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3833154681"}]: dispatch 2024-03-12T03:43:57.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2623870350' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3833154681"}]': finished 2024-03-12T03:43:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[24773]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:43:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/7903998' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]: dispatch 2024-03-12T03:43:57.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]: dispatch 2024-03-12T03:43:57.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2623870350' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3833154681"}]': finished 2024-03-12T03:43:57.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[29473]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:43:57.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/7903998' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]: dispatch 2024-03-12T03:43:57.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:57 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]: dispatch 2024-03-12T03:43:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:57 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2623870350' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3833154681"}]': finished 2024-03-12T03:43:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:57 smithi153 ceph-mon[25643]: osdmap e99: 8 total, 8 up, 8 in 2024-03-12T03:43:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:57 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/7903998' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]: dispatch 2024-03-12T03:43:57.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:57 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]: dispatch 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]': finished 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[24773]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[24773]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3880210845' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]: dispatch 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]: dispatch 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]': finished 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[29473]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[29473]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3880210845' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]: dispatch 2024-03-12T03:43:58.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:58 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]: dispatch 2024-03-12T03:43:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:58 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2748554269"}]': finished 2024-03-12T03:43:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:58 smithi153 ceph-mon[25643]: osdmap e100: 8 total, 8 up, 8 in 2024-03-12T03:43:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:58 smithi153 ceph-mon[25643]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:43:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:58 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3880210845' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]: dispatch 2024-03-12T03:43:58.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:58 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]: dispatch 2024-03-12T03:43:59.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:59 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]': finished 2024-03-12T03:43:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:59 smithi098 ceph-mon[24773]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:43:59.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:43:59 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2265990785' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4044504336"}]: dispatch 2024-03-12T03:43:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:59 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]': finished 2024-03-12T03:43:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:59 smithi098 ceph-mon[29473]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:43:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:43:59 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2265990785' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4044504336"}]: dispatch 2024-03-12T03:43:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:59 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1273287586"}]': finished 2024-03-12T03:43:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:59 smithi153 ceph-mon[25643]: osdmap e101: 8 total, 8 up, 8 in 2024-03-12T03:43:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:43:59 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2265990785' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4044504336"}]: dispatch 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2265990785' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4044504336"}]': finished 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[24773]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[24773]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1597211468' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]: dispatch 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]: dispatch 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2265990785' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4044504336"}]': finished 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[29473]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[29473]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:00.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1597211468' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]: dispatch 2024-03-12T03:44:00.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:00 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]: dispatch 2024-03-12T03:44:00.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:44:00] "GET /metrics HTTP/1.1" 200 37895 "" "Prometheus/2.33.4" 2024-03-12T03:44:00.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:00 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2265990785' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/4044504336"}]': finished 2024-03-12T03:44:00.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:00 smithi153 ceph-mon[25643]: osdmap e102: 8 total, 8 up, 8 in 2024-03-12T03:44:00.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:00 smithi153 ceph-mon[25643]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:00.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:00 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1597211468' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]: dispatch 2024-03-12T03:44:00.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:00 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]: dispatch 2024-03-12T03:44:01.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:01 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]': finished 2024-03-12T03:44:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:01 smithi098 ceph-mon[24773]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:44:01.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:01 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]': finished 2024-03-12T03:44:01.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:01 smithi098 ceph-mon[29473]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:44:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:01 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/1958076942"}]': finished 2024-03-12T03:44:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:01 smithi153 ceph-mon[25643]: osdmap e103: 8 total, 8 up, 8 in 2024-03-12T03:44:02.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:02 smithi098 ceph-mon[24773]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:02.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:02 smithi098 ceph-mon[29473]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:02.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:02 smithi153 ceph-mon[25643]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:03.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:03 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:03 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:03 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:04.840 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:04 smithi153 ceph-mon[25643]: from='client.34817 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:04.840 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:04 smithi153 ceph-mon[25643]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:04.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:04 smithi098 ceph-mon[24773]: from='client.34817 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:04.967 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:04 smithi098 ceph-mon[24773]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:04.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:04 smithi098 ceph-mon[29473]: from='client.34817 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:04.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:04 smithi098 ceph-mon[29473]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:06 smithi098 ceph-mon[24773]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T03:44:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:06 smithi098 ceph-mon[29473]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T03:44:07.100 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:06 smithi153 ceph-mon[25643]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T03:44:08.924 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:08 smithi153 ceph-mon[25643]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-03-12T03:44:08.924 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:08 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:08 smithi098 ceph-mon[24773]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-03-12T03:44:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:08 smithi098 ceph-mon[29473]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 629 B/s rd, 0 op/s 2024-03-12T03:44:10.744 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:44:10] "GET /metrics HTTP/1.1" 200 37891 "" "Prometheus/2.33.4" 2024-03-12T03:44:11.071 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:10 smithi153 ceph-mon[25643]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:44:11.072 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:10 smithi153 systemd[1]: Stopping Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:44:11.072 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:10 smithi153 bash[49588]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana.a" found: no such container 2024-03-12T03:44:11.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:10 smithi098 ceph-mon[24773]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:44:11.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:10 smithi098 ceph-mon[29473]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:44:11.357 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:44:11+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-12T03:44:11.357 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[41757]: t=2024-03-12T03:44:11+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-03-12T03:44:11.357 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 podman[49601]: 2024-03-12 03:44:11.090515069 +0000 UTC m=+0.138478455 container died 7864a9bab9e4e38bf5319a56ba0d772222b3e201c284919cefc5136983aac81a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, maintainer=Paul Cuzner , name=ubi8, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, version=8.5, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public) 2024-03-12T03:44:11.693 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 podman[49601]: 2024-03-12 03:44:11.444612962 +0000 UTC m=+0.492576346 container cleanup 7864a9bab9e4e38bf5319a56ba0d772222b3e201c284919cefc5136983aac81a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, release=236.1648460182, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, description=Ceph Grafana Container, distribution-scope=public, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-12T03:44:11.693 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 bash[49601]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a 2024-03-12T03:44:11.693 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 bash[49628]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana.a" found: no such container 2024-03-12T03:44:12.029 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 podman[49615]: 2024-03-12 03:44:11.695692809 +0000 UTC m=+0.605327392 container remove 7864a9bab9e4e38bf5319a56ba0d772222b3e201c284919cefc5136983aac81a (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, architecture=x86_64, vcs-type=git, com.redhat.component=ubi8-container, distribution-scope=public, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, description=Ceph Grafana Container, io.buildah.version=1.24.2, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, io.openshift.tags=base rhel8) 2024-03-12T03:44:12.029 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-12T03:44:12.030 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 systemd[1]: Stopped Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:44:12.030 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service: Consumed 3.347s CPU time. 2024-03-12T03:44:12.030 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 systemd[1]: Starting Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:44:12.030 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 podman[49698]: 2024-03-12 03:44:11.930043188 +0000 UTC m=+0.110758728 container create 9057d202c6a75776c0a39a39c0fe4005fccfbed59393ce506c5875774bf52907 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, vendor=Red Hat, Inc., io.buildah.version=1.24.2, release=236.1648460182, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-type=git, maintainer=Paul Cuzner , distribution-scope=public, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, description=Ceph Grafana Container) 2024-03-12T03:44:12.030 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:11 smithi153 podman[49698]: 2024-03-12 03:44:11.836855078 +0000 UTC m=+0.017570626 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-12T03:44:12.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STOPPING 2024-03-12T03:44:12.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:44:12.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STOPPED 2024-03-12T03:44:12.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STARTING 2024-03-12T03:44:12.443 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 podman[49698]: 2024-03-12 03:44:12.12382577 +0000 UTC m=+0.304541317 container init 9057d202c6a75776c0a39a39c0fe4005fccfbed59393ce506c5875774bf52907 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, build-date=2022-03-28T10:36:18.413762, name=ubi8, vendor=Red Hat, Inc., version=8.5, architecture=x86_64, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, description=Ceph Grafana Container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, io.openshift.expose-services=, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 podman[49698]: 2024-03-12 03:44:12.12644842 +0000 UTC m=+0.307163969 container start 9057d202c6a75776c0a39a39c0fe4005fccfbed59393ce506c5875774bf52907 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, io.openshift.expose-services=, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, release=236.1648460182, architecture=x86_64, name=ubi8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762) 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-12T03:44:12.444 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="App mode production" logger=settings 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=526.632µs 2024-03-12T03:44:12.445 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 bash[49698]: 9057d202c6a75776c0a39a39c0fe4005fccfbed59393ce506c5875774bf52907 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 systemd[1]: Started Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-12T03:44:12.446 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-12T03:44:12.447 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:44:12+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-12T03:44:12.746 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Serving on http://:::9283 2024-03-12T03:44:12.746 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STARTED 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STOPPING 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STOPPED 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STARTING 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Serving on http://:::9283 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STARTED 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STOPPING 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STOPPED 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STARTING 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Serving on http://:::9283 2024-03-12T03:44:12.747 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:44:12] ENGINE Bus STARTED 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:44:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:44:13.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:44:13.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:12 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.121 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T03:44:13.121 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.121 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.121 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:44:13.122 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:12 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:13.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:13 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:44:13.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:13 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:44:13.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:13 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:44:13.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:13 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:44:13.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:13 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:44:13.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:13 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:44:13.844 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:13 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:14.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:44:14.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:44:14.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:44:14.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:44:14.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:44:14.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:44:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:44:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi098.front.sepia.ceph.com:9093"}]: dispatch 2024-03-12T03:44:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:44:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-03-12T03:44:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:44:14.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:13 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-03-12T03:44:14.999 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:14 smithi098 ceph-mon[29473]: from='client.34817 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:14.999 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:14 smithi098 ceph-mon[29473]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:15.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:14 smithi098 ceph-mon[24773]: from='client.34817 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:15.000 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:14 smithi098 ceph-mon[24773]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:14 smithi153 ceph-mon[25643]: from='client.34817 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:14 smithi153 ceph-mon[25643]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:44:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:44:17.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:16 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:44:17.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:16 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:44:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:17 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:44:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:17 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:44:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:17 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:17 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:44:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:17 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:44:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:17 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:18.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:17 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:44:18.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:17 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:44:18.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:17 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:18.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:19.247 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:18 smithi098 ceph-mon[24773]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:19.247 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:18 smithi098 ceph-mon[29473]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:19.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:18 smithi153 ceph-mon[25643]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:19.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:44:19.513+0000 7f056598d700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:19.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Creating ceph-iscsi config... 2024-03-12T03:44:19.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Traceback (most recent call last): 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return _run_code(code, main_globals, None, 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: exec(code, run_globals) 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:19.944 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Traceback (most recent call last): 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:44:19.945 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return future.result(timeout) 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return self.__get_result() 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: raise self._exception 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: config_blobs=daemon_spec.final_config, 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Creating ceph-iscsi config... 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:19.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: Traceback (most recent call last): 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: return _run_code(code, main_globals, None, 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: exec(code, run_globals) 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:19.947 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:19.948 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:19.948 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:19.948 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:19.948 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:19.948 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:19.948 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:19.948 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:20.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:44:20] "GET /metrics HTTP/1.1" 200 37889 "" "Prometheus/2.33.4" 2024-03-12T03:44:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:20 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:44:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:21.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:44:21.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:21.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:21.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:21.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:20 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:21.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:44:21.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:21.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:21.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:21.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:21.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:44:21.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:21.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:21.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:20 smithi098 ceph-mon[29473]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:20 smithi098 ceph-mon[24773]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:20 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:44:21.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:44:21.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:44:21.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:44:21.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:44:21.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:44:21.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:44:21.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:44:21.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:44:21.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:44:21.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:20 smithi153 ceph-mon[25643]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:23.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:22 smithi098 ceph-mon[24773]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:22 smithi098 ceph-mon[29473]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:22 smithi153 ceph-mon[25643]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:23.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:24.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:24 smithi098 ceph-mon[24773]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:24.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:24 smithi098 ceph-mon[29473]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:24 smithi153 ceph-mon[25643]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:27.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:26 smithi098 ceph-mon[24773]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:27.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:26 smithi098 ceph-mon[29473]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:26 smithi153 ceph-mon[25643]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:28 smithi153 ceph-mon[25643]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:44:28.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:28 smithi098 ceph-mon[24773]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:44:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:28 smithi098 ceph-mon[29473]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:44:30.751 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:44:30] "GET /metrics HTTP/1.1" 200 37889 "" "Prometheus/2.33.4" 2024-03-12T03:44:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:30 smithi098 ceph-mon[24773]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:44:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:30 smithi098 ceph-mon[29473]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:44:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:30 smithi153 ceph-mon[25643]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:44:32.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:31 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3518359106' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:44:32.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:31 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3518359106' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:44:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:31 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3518359106' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:44:33.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:32 smithi098 ceph-mon[24773]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:44:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:32 smithi098 ceph-mon[29473]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:44:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:32 smithi153 ceph-mon[25643]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:44:33.761 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:34.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:33 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:33 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:34.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:33 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:34 smithi098 ceph-mon[24773]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:44:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:34 smithi098 ceph-mon[29473]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:44:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:34 smithi153 ceph-mon[25643]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:44:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:36 smithi098 ceph-mon[24773]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:44:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:36 smithi098 ceph-mon[29473]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:44:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:36 smithi153 ceph-mon[25643]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:44:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:38 smithi153 ceph-mon[25643]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:44:38.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:38 smithi098 ceph-mon[24773]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:44:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:38 smithi098 ceph-mon[29473]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:44:40.773 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:44:40] "GET /metrics HTTP/1.1" 200 37889 "" "Prometheus/2.33.4" 2024-03-12T03:44:41.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:40 smithi098 ceph-mon[24773]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:40 smithi098 ceph-mon[29473]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:40 smithi153 ceph-mon[25643]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:42.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:41 smithi098 ceph-mon[24773]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:42.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:41 smithi098 ceph-mon[29473]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:42.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:41 smithi153 ceph-mon[25643]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:43.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:42 smithi098 ceph-mon[24773]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:42 smithi098 ceph-mon[29473]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:42 smithi153 ceph-mon[25643]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:43.947 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:44 smithi098 ceph-mon[24773]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:44 smithi098 ceph-mon[29473]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:44 smithi153 ceph-mon[25643]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:44:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:46 smithi098 ceph-mon[24773]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:44:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:46 smithi098 ceph-mon[29473]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:44:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:46 smithi153 ceph-mon[25643]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:44:48.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:48.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:48 smithi153 ceph-mon[25643]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:48.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:48 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:48 smithi098 ceph-mon[24773]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:48 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:48 smithi098 ceph-mon[29473]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:48 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:44:50.753 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:44:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:44:50] "GET /metrics HTTP/1.1" 200 37889 "" "Prometheus/2.33.4" 2024-03-12T03:44:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:50 smithi098 ceph-mon[24773]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:50 smithi098 ceph-mon[29473]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:50 smithi153 ceph-mon[25643]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:52.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:51 smithi098 ceph-mon[24773]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:51 smithi098 ceph-mon[29473]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:52.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:51 smithi153 ceph-mon[25643]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:44:53.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:52 smithi098 ceph-mon[24773]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:52 smithi098 ceph-mon[29473]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:53.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:52 smithi153 ceph-mon[25643]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:53.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:54.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:54 smithi098 ceph-mon[24773]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:54 smithi098 ceph-mon[29473]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:54 smithi153 ceph-mon[25643]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:56 smithi098 ceph-mon[29473]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:56 smithi098 ceph-mon[24773]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:56 smithi153 ceph-mon[25643]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:44:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:44:58 smithi153 ceph-mon[25643]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:44:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:44:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:44:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:44:58 smithi098 ceph-mon[24773]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:44:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:44:58 smithi098 ceph-mon[29473]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:00.759 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:45:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:45:00] "GET /metrics HTTP/1.1" 200 37889 "" "Prometheus/2.33.4" 2024-03-12T03:45:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:00 smithi098 ceph-mon[24773]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:01.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:00 smithi098 ceph-mon[29473]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:01.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:00 smithi153 ceph-mon[25643]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:02.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:01 smithi098 ceph-mon[24773]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:02.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:01 smithi098 ceph-mon[29473]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:02.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:01 smithi153 ceph-mon[25643]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:02 smithi098 ceph-mon[24773]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:02 smithi098 ceph-mon[29473]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:02 smithi153 ceph-mon[25643]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:03.767 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:04.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:03 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:04.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:03 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:04.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:03 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:05.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:04 smithi098 ceph-mon[24773]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:04 smithi098 ceph-mon[29473]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:04 smithi153 ceph-mon[25643]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:06 smithi098 ceph-mon[24773]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:06 smithi098 ceph-mon[29473]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:07.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:06 smithi153 ceph-mon[25643]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:08 smithi153 ceph-mon[25643]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:08.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:08 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:09.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:08 smithi098 ceph-mon[24773]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:08 smithi098 ceph-mon[29473]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:10.780 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:45:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:45:10] "GET /metrics HTTP/1.1" 200 37884 "" "Prometheus/2.33.4" 2024-03-12T03:45:11.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:10 smithi098 ceph-mon[24773]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:11.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:10 smithi098 ceph-mon[29473]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:10 smithi153 ceph-mon[25643]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:12.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:11 smithi098 ceph-mon[24773]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:12.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:11 smithi098 ceph-mon[29473]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:12.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:11 smithi153 ceph-mon[25643]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:13.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:12 smithi098 ceph-mon[24773]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:12 smithi098 ceph-mon[29473]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:12 smithi153 ceph-mon[25643]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:13.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:13 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:14.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:14 smithi098 ceph-mon[24773]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:14.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:14 smithi098 ceph-mon[29473]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:14 smithi153 ceph-mon[25643]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:16 smithi098 ceph-mon[24773]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:16 smithi098 ceph-mon[29473]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:16 smithi153 ceph-mon[25643]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:18 smithi153 ceph-mon[25643]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:19.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:18 smithi098 ceph-mon[29473]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:19.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:19.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:18 smithi098 ceph-mon[24773]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:19.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:20.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:45:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:45:20] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-03-12T03:45:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:20 smithi098 ceph-mon[24773]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:20 smithi098 ceph-mon[29473]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:20 smithi153 ceph-mon[25643]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:21 smithi153 ceph-mon[25643]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:22.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:21 smithi098 ceph-mon[24773]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:21 smithi098 ceph-mon[29473]: from='client.25066 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:22.996 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:22 smithi098 ceph-mon[24773]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:22.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:22 smithi098 ceph-mon[29473]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:23.123 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:22 smithi153 ceph-mon[25643]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:23.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:23.503Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:45:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:45:24.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:24 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:45:24.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:24 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:25.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:25 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:45:25.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:25 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:45:25.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:25 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:45:25.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:25 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:45:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:25 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:45:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:25 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:45:26.686 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:26 smithi098 ceph-mon[24773]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:26.686 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:26 smithi098 ceph-mon[29473]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:26 smithi153 ceph-mon[25643]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:28.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:28 smithi153 ceph-mon[25643]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:28 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:45:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:28 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:45:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:28 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2500050950' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:45:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[24773]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:45:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:45:29.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2500050950' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:45:29.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[29473]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:29.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:29.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:29.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:45:29.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:45:29.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:28 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2500050950' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:45:30.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:29 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:45:30.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:29 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:45:30.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:29 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:45:30.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:29 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:45:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:29 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:45:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:29 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:45:30.765 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:45:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:45:30] "GET /metrics HTTP/1.1" 200 37885 "" "Prometheus/2.33.4" 2024-03-12T03:45:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:30 smithi098 ceph-mon[24773]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:30 smithi098 ceph-mon[29473]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:30 smithi153 ceph-mon[25643]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:33 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:45:33.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:33.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:34.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:34 smithi098 ceph-mon[24773]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:34.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:34 smithi098 ceph-mon[29473]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:34 smithi153 ceph-mon[25643]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:36 smithi098 ceph-mon[24773]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:36 smithi098 ceph-mon[29473]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:36 smithi153 ceph-mon[25643]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:38 smithi153 ceph-mon[25643]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:38.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:39.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:38 smithi098 ceph-mon[24773]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:38 smithi098 ceph-mon[29473]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:45:40.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:39 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:40.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:39 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:40.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:39 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:40.772 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:45:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:45:40] "GET /metrics HTTP/1.1" 200 37897 "" "Prometheus/2.33.4" 2024-03-12T03:45:41.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:40 smithi098 ceph-mon[24773]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:40 smithi098 ceph-mon[29473]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:40 smithi153 ceph-mon[25643]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:43.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:42 smithi098 ceph-mon[24773]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:42 smithi098 ceph-mon[29473]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:42 smithi153 ceph-mon[25643]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:43.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:44 smithi098 ceph-mon[24773]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:44 smithi098 ceph-mon[29473]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:44 smithi153 ceph-mon[25643]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:46 smithi098 ceph-mon[24773]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:46 smithi098 ceph-mon[29473]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:46 smithi153 ceph-mon[25643]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:48.692 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:48 smithi098 ceph-mon[24773]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:48 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:48 smithi098 ceph-mon[29473]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:48 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:48 smithi153 ceph-mon[25643]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:49.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:48 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:45:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:49 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:50.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:49 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:49 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:45:50.779 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:45:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:45:50] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-03-12T03:45:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:50 smithi098 ceph-mon[24773]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:50 smithi098 ceph-mon[29473]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:50 smithi153 ceph-mon[25643]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:53.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:52 smithi098 ceph-mon[24773]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:52 smithi098 ceph-mon[29473]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:53.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:52 smithi153 ceph-mon[25643]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:53.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:54.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:54 smithi098 ceph-mon[24773]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:54 smithi098 ceph-mon[29473]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:54 smithi153 ceph-mon[25643]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:45:57.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:56 smithi098 ceph-mon[24773]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:56 smithi098 ceph-mon[29473]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:56 smithi153 ceph-mon[25643]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:58 smithi153 ceph-mon[25643]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:45:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:45:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:45:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:58 smithi098 ceph-mon[29473]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:45:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:58 smithi098 ceph-mon[24773]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:45:59 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:45:59 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:00.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:45:59 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:00.778 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:46:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:46:00] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-03-12T03:46:01.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:00 smithi098 ceph-mon[29473]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:00 smithi098 ceph-mon[24773]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:01.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:00 smithi153 ceph-mon[25643]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:02 smithi098 ceph-mon[29473]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:02 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:02 smithi098 ceph-mon[24773]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:02 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:02 smithi153 ceph-mon[25643]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:03.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:02 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:03.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:04.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:04 smithi098 ceph-mon[24773]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:04.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:04 smithi098 ceph-mon[29473]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:04.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:04 smithi153 ceph-mon[25643]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:06 smithi098 ceph-mon[24773]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:06 smithi098 ceph-mon[29473]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:07.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:06 smithi153 ceph-mon[25643]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:08 smithi153 ceph-mon[25643]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:08.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:08 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:08 smithi098 ceph-mon[29473]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:08 smithi098 ceph-mon[24773]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:10.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:09 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:09 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:09 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:10.783 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:46:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:46:10] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-03-12T03:46:11.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:10 smithi098 ceph-mon[29473]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:11.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:10 smithi098 ceph-mon[24773]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:10 smithi153 ceph-mon[25643]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:13.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:12 smithi098 ceph-mon[24773]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:12 smithi098 ceph-mon[29473]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:12 smithi153 ceph-mon[25643]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:13.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:13 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:14 smithi098 ceph-mon[24773]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:14.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:14 smithi098 ceph-mon[29473]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:14 smithi153 ceph-mon[25643]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:16 smithi098 ceph-mon[24773]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:16 smithi098 ceph-mon[29473]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:16 smithi153 ceph-mon[25643]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:18.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:18 smithi153 ceph-mon[25643]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:18 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:19.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:18 smithi098 ceph-mon[24773]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:19.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:18 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:18 smithi098 ceph-mon[29473]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:18 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:19 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:19 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:19 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:20.574 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-12T03:46:20.788 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:46:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:46:20] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-03-12T03:46:20.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:20 smithi098 ceph-mon[24773]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:20.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:20 smithi098 ceph-mon[29473]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:20 smithi153 ceph-mon[25643]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:22.755 INFO:teuthology.orchestra.run.smithi098.stdout:Scheduled to redeploy mgr.y on host 'smithi098' 2024-03-12T03:46:23.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[24773]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1186659735' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[29473]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1186659735' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:22 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:22 smithi153 ceph-mon[25643]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:22 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1186659735' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-12T03:46:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:22 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:22 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:22 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:22 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:22 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:46:23.392 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps --refresh' 2024-03-12T03:46:23.800 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:23.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:46:23.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: Schedule redeploy daemon mgr.y 2024-03-12T03:46:23.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:46:23.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[24773]: Deploying daemon mgr.y on smithi098 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: Schedule redeploy daemon mgr.y 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:23.952 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:23 smithi098 ceph-mon[29473]: Deploying daemon mgr.y on smithi098 2024-03-12T03:46:24.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: from='client.15207 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:46:24.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: Schedule redeploy daemon mgr.y 2024-03-12T03:46:24.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:24.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:46:24.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:46:24.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:46:24.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:46:24.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:23 smithi153 ceph-mon[25643]: Deploying daemon mgr.y on smithi098 2024-03-12T03:46:24.911 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:24 smithi098 ceph-mon[24773]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:24.912 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:24 smithi098 ceph-mon[29473]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:24 smithi153 ceph-mon[25643]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (2m) 53s ago 10m 17.7M - 0.23.0 ba2b418f427c ae9380406106 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (2m) 53s ago 10m 50.0M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (57s) 53s ago 9m 47.8M - 3.5 e1d6a67b021e bd06cc3b03b7 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283 running (5m) 53s ago 15m 505M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 2e7f2996e113 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:9283 running (16m) 53s ago 16m 410M - 17.2.0 e1d6a67b021e 3c66c3a9135b 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (16m) 53s ago 16m 55.8M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (15m) 53s ago 15m 46.0M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (15m) 53s ago 15m 47.2M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (2m) 53s ago 11m 16.7M - 1.3.1 1dbe0e931976 ff35fa2e462e 2024-03-12T03:46:25.482 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (2m) 53s ago 11m 9298k - 1.3.1 1dbe0e931976 78fb1d60e454 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (14m) 53s ago 14m 53.9M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (14m) 53s ago 14m 55.5M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (13m) 53s ago 13m 49.0M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (13m) 53s ago 13m 51.3M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (13m) 53s ago 13m 55.0M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (12m) 53s ago 12m 52.3M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (12m) 53s ago 12m 48.8M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (12m) 53s ago 12m 51.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (2m) 53s ago 11m 63.2M - 2.33.4 514e6a882f6e 8b54432f2054 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (9m) 53s ago 9m 72.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:46:25.483 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (9m) 53s ago 9m 75.4M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:46:26.311 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'sleep 180' 2024-03-12T03:46:26.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:26 smithi098 ceph-mon[24773]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:46:26.992 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:26 smithi098 ceph-mon[24773]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:26.992 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:26 smithi098 ceph-mon[29473]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:46:26.992 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:26 smithi098 ceph-mon[29473]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:26 smithi153 ceph-mon[25643]: from='client.15213 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:46:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:26 smithi153 ceph-mon[25643]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:28 smithi153 ceph-mon[25643]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:28.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:28.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:29.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:28 smithi098 ceph-mon[24773]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:28 smithi098 ceph-mon[29473]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:30.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:29 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:30.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:29 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:29 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:30.814 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:46:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:46:30] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-03-12T03:46:31.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:30 smithi098 ceph-mon[24773]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:30 smithi098 ceph-mon[29473]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:30 smithi153 ceph-mon[25643]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:33.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:32 smithi098 ceph-mon[24773]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:33.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:32 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:32 smithi098 ceph-mon[29473]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:32 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:32 smithi153 ceph-mon[25643]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:33.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:32 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:33.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:34.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:34 smithi098 ceph-mon[24773]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:34.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:34 smithi098 ceph-mon[29473]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:34 smithi153 ceph-mon[25643]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:46:37.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:36 smithi098 ceph-mon[24773]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:36 smithi098 ceph-mon[29473]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:36 smithi153 ceph-mon[25643]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:38.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:38 smithi153 ceph-mon[25643]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:38.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:39.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:38 smithi098 ceph-mon[24773]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:38 smithi098 ceph-mon[29473]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:40.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:39 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:40.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:39 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:40.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:39 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:40.788 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:46:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:46:40] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-03-12T03:46:41.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:40 smithi098 ceph-mon[24773]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:40 smithi098 ceph-mon[29473]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:40 smithi153 ceph-mon[25643]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:43.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:42 smithi098 ceph-mon[24773]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:42 smithi098 ceph-mon[29473]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:42 smithi153 ceph-mon[25643]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:43.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:44.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:44 smithi098 ceph-mon[24773]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:46:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:44 smithi098 ceph-mon[29473]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:46:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:44 smithi153 ceph-mon[25643]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:46:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:46 smithi098 ceph-mon[24773]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:46:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:46 smithi098 ceph-mon[29473]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:46:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:46 smithi153 ceph-mon[25643]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:46:48.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:47 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:47 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:47 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:46:48.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:48 smithi153 ceph-mon[25643]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:46:48.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:48 smithi098 ceph-mon[24773]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:46:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:48 smithi098 ceph-mon[29473]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:46:50.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:49 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:50.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:49 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:49 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:46:50.799 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:46:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:46:50] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-03-12T03:46:51.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:50 smithi098 ceph-mon[24773]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:46:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:50 smithi098 ceph-mon[29473]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:46:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:50 smithi153 ceph-mon[25643]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:46:53.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:52 smithi098 ceph-mon[24773]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:52 smithi098 ceph-mon[29473]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:53.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:52 smithi153 ceph-mon[25643]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:53.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:53 smithi098 systemd[1]: Stopping Ceph mgr.y for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:46:53.896 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:53 smithi098 podman[62868]: 2024-03-12 03:46:53.591375873 +0000 UTC m=+0.137526257 container died 3c66c3a9135ba656713d1d24064af4437be41c7111ac7df6289ffa8277d0b847 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, ceph=True, name=centos-stream, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , architecture=x86_64, release=754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-12T03:46:53.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:54.188 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:53 smithi098 podman[62868]: 2024-03-12 03:46:53.969035355 +0000 UTC m=+0.515185740 container cleanup 3c66c3a9135ba656713d1d24064af4437be41c7111ac7df6289ffa8277d0b847 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, vcs-type=git, version=8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_CLEAN=True) 2024-03-12T03:46:54.188 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:53 smithi098 bash[62868]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y 2024-03-12T03:46:54.522 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 podman[62882]: 2024-03-12 03:46:54.187036454 +0000 UTC m=+0.596695765 container remove 3c66c3a9135ba656713d1d24064af4437be41c7111ac7df6289ffa8277d0b847 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.openshift.expose-services=, release=754, io.buildah.version=1.19.8) 2024-03-12T03:46:54.523 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:46:54.523 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-12T03:46:54.523 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 systemd[1]: Stopped Ceph mgr.y for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:46:54.523 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service: Consumed 1min 865ms CPU time. 2024-03-12T03:46:54.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:54 smithi098 ceph-mon[24773]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:54 smithi098 ceph-mon[29473]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:54.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 systemd[1]: Starting Ceph mgr.y for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:46:54.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 podman[63010]: 2024-03-12 03:46:54.638208989 +0000 UTC m=+0.018363625 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:46:54.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 podman[63010]: 2024-03-12 03:46:54.75520364 +0000 UTC m=+0.135358275 container create 20b96dbb429c2dc568d5fdd15316472120f89ddfb1ba5d4ac4b27ea80ed73cc1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-12T03:46:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:54 smithi153 ceph-mon[25643]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-12T03:46:55.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 podman[63010]: 2024-03-12 03:46:54.932584327 +0000 UTC m=+0.312738973 container init 20b96dbb429c2dc568d5fdd15316472120f89ddfb1ba5d4ac4b27ea80ed73cc1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-12T03:46:55.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 podman[63010]: 2024-03-12 03:46:54.935189145 +0000 UTC m=+0.315343790 container start 20b96dbb429c2dc568d5fdd15316472120f89ddfb1ba5d4ac4b27ea80ed73cc1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T03:46:55.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:54 smithi098 ceph-mgr[63028]: -- 172.21.15.98:0/3634757835 <== mon.1 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564ee094e5a0 con 0x564ee0922800 2024-03-12T03:46:55.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:55 smithi098 bash[63010]: 20b96dbb429c2dc568d5fdd15316472120f89ddfb1ba5d4ac4b27ea80ed73cc1 2024-03-12T03:46:55.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:55 smithi098 systemd[1]: Started Ceph mgr.y for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:46:55.636 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:55.523+0000 7f29b7634200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:46:55.636 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:55.633+0000 7f29b7634200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:46:56.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[24773]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-mon[29473]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:56.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:56 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:56 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:56 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:56 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:46:56.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:56 smithi153 ceph-mon[25643]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:56.593 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:56.322+0000 7f29b7634200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:46:56.593 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:56.443+0000 7f29b7634200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:46:57.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:56.590+0000 7f29b7634200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:46:57.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:56.692+0000 7f29b7634200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:46:57.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:56.789+0000 7f29b7634200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:46:57.074 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:57.070+0000 7f29b7634200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:46:57.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:57.193+0000 7f29b7634200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:46:57.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:57.297+0000 7f29b7634200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:46:57.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:57.399+0000 7f29b7634200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:46:57.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:57.587+0000 7f29b7634200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:46:58.789 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:58 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:58.596+0000 7f29b7634200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:46:58.789 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:58 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:58.737+0000 7f29b7634200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:46:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:58 smithi153 ceph-mon[25643]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:46:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:46:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:46:59.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:58 smithi098 ceph-mon[24773]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:58 smithi098 ceph-mon[29473]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:46:59.400 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:59.079+0000 7f29b7634200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:46:59.400 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:59.192+0000 7f29b7634200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:46:59.400 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:59.397+0000 7f29b7634200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:46:59.790 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:59.594+0000 7f29b7634200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:47:00.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:46:59.888+0000 7f29b7634200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:47:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:00.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:00.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:46:59 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:00.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:59 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:00.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:59 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:00.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:46:59 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:00.423 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:47:00.162+0000 7f29b7634200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:47:00.423 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:47:00.420+0000 7f29b7634200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:47:00.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:47:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:47:00] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-03-12T03:47:01.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[24773]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:01.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:01.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:01.251 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.252 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[29473]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:01.252 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.252 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.252 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:01.252 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:01.252 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:00 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:00 smithi153 ceph-mon[25643]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:00 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:00 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:00 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:47:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:00 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:47:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:00 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:47:01.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:47:01.248+0000 7f29b7634200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:47:02.155 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:47:01.831+0000 7f29b7634200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:47:02.156 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:47:01.931+0000 7f29b7634200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:47:02.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:47:02.153+0000 7f29b7634200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:47:02.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:47:02] ENGINE Bus STARTING 2024-03-12T03:47:02.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CherryPy Checker: 2024-03-12T03:47:02.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: The Application mounted at '' has an empty config. 2024-03-12T03:47:02.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:47:02.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:47:02] ENGINE Serving on http://:::9283 2024-03-12T03:47:02.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:47:02] ENGINE Bus STARTED 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: Standby manager daemon y restarted 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: Standby manager daemon y started 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:03.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:03.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: Standby manager daemon y restarted 2024-03-12T03:47:03.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: Standby manager daemon y started 2024-03-12T03:47:03.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:47:03.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:47:03.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:47:03.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:47:03.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:02 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: Standby manager daemon y restarted 2024-03-12T03:47:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: Standby manager daemon y started 2024-03-12T03:47:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:47:03.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:47:03.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:47:03.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:47:03.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:02 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:03.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:04.245 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:03 smithi098 ceph-mon[24773]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-12T03:47:04.246 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:03 smithi098 ceph-mon[29473]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-12T03:47:04.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:03 smithi153 ceph-mon[25643]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-12T03:47:05.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:04 smithi098 ceph-mon[24773]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:05.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:04 smithi098 ceph-mon[29473]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:05.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:04 smithi153 ceph-mon[25643]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:07.244 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:06 smithi153 ceph-mon[25643]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:06 smithi098 ceph-mon[24773]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:06 smithi098 ceph-mon[29473]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:08.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:08 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:09.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:08 smithi098 ceph-mon[24773]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:09.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:08 smithi098 ceph-mon[29473]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:09.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:08 smithi153 ceph-mon[25643]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:10.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:09 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:10.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:09 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:10.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:09 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:10.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:47:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:47:10] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-03-12T03:47:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:10 smithi098 ceph-mon[24773]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:11.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:10 smithi098 ceph-mon[29473]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:11 smithi153 ceph-mon[25643]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:13.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:13 smithi098 ceph-mon[24773]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:13.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:13 smithi098 ceph-mon[29473]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:13.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:13 smithi153 ceph-mon[25643]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:13.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:13 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:14 smithi098 ceph-mon[24773]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:14.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:14 smithi098 ceph-mon[29473]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:14 smithi153 ceph-mon[25643]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:17.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:16 smithi098 ceph-mon[24773]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:16 smithi098 ceph-mon[29473]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:16 smithi153 ceph-mon[25643]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:18.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:17 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:17 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:18.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:17 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:18.801 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:18 smithi098 ceph-mon[29473]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:19.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:18 smithi098 ceph-mon[24773]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:19.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:18 smithi153 ceph-mon[25643]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:19 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:19 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:19 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:20.811 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:47:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:47:20] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-03-12T03:47:21.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:20 smithi098 ceph-mon[24773]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:20 smithi098 ceph-mon[29473]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:20 smithi153 ceph-mon[25643]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:22 smithi098 ceph-mon[24773]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:22 smithi098 ceph-mon[29473]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:22 smithi153 ceph-mon[25643]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:23.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:24.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:24 smithi098 ceph-mon[24773]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:24.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:24 smithi098 ceph-mon[29473]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:24 smithi153 ceph-mon[25643]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:27.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:26 smithi098 ceph-mon[24773]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:27.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:26 smithi098 ceph-mon[29473]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:26 smithi153 ceph-mon[25643]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:28 smithi153 ceph-mon[25643]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:28.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:28 smithi098 ceph-mon[24773]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:28 smithi098 ceph-mon[29473]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:30.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:29 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:30.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:29 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:29 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:30.806 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:47:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:47:30] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-03-12T03:47:31.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:30 smithi098 ceph-mon[24773]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:30 smithi098 ceph-mon[29473]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:30 smithi153 ceph-mon[25643]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:33.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:32 smithi098 ceph-mon[24773]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:33.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:32 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:32 smithi098 ceph-mon[29473]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:32 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:32 smithi153 ceph-mon[25643]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:32 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:33.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:34.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:34 smithi098 ceph-mon[24773]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:34.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:34 smithi098 ceph-mon[29473]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:34 smithi153 ceph-mon[25643]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:36 smithi098 ceph-mon[24773]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:36 smithi098 ceph-mon[29473]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:36 smithi153 ceph-mon[25643]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:38.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:38 smithi153 ceph-mon[25643]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:38.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:39.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:38 smithi098 ceph-mon[24773]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:38 smithi098 ceph-mon[29473]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:40.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:39 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:40.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:39 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:40.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:39 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:40.808 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:47:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:47:40] "GET /metrics HTTP/1.1" 200 37900 "" "Prometheus/2.33.4" 2024-03-12T03:47:41.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:40 smithi098 ceph-mon[24773]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:40 smithi098 ceph-mon[29473]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:40 smithi153 ceph-mon[25643]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:43.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:42 smithi098 ceph-mon[24773]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:42 smithi098 ceph-mon[29473]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:42 smithi153 ceph-mon[25643]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:43.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:43.506Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:44.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:44 smithi098 ceph-mon[24773]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:44.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:44 smithi098 ceph-mon[29473]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:44 smithi153 ceph-mon[25643]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:46 smithi098 ceph-mon[24773]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:46 smithi098 ceph-mon[29473]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:46 smithi153 ceph-mon[25643]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:48.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:47 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:48.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:47 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:47 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:47:48.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:48 smithi153 ceph-mon[25643]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:48.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:49.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:48 smithi098 ceph-mon[29473]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:48 smithi098 ceph-mon[24773]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:49 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:50.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:49 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:49 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:47:50.821 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:47:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:47:50] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-03-12T03:47:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:50 smithi098 ceph-mon[24773]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:50 smithi098 ceph-mon[29473]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:50 smithi153 ceph-mon[25643]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:53.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:52 smithi153 ceph-mon[25643]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:53.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:52 smithi098 ceph-mon[24773]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:53.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:52 smithi098 ceph-mon[29473]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:53.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:54 smithi153 ceph-mon[25643]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:55.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:54 smithi098 ceph-mon[24773]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:54 smithi098 ceph-mon[29473]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:47:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:56 smithi098 ceph-mon[24773]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:56 smithi098 ceph-mon[29473]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:56 smithi153 ceph-mon[25643]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:58.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:58 smithi153 ceph-mon[25643]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:58.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:47:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:47:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:47:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:58 smithi098 ceph-mon[24773]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:47:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:58 smithi098 ceph-mon[29473]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:00.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:47:59 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:47:59 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:00.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:47:59 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:00.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:48:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:48:00] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-03-12T03:48:01.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:00 smithi098 ceph-mon[24773]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:01.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:00 smithi098 ceph-mon[29473]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:01.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:00 smithi153 ceph-mon[25643]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:03.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:02 smithi098 ceph-mon[24773]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:02 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:02 smithi098 ceph-mon[29473]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:02 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:02 smithi153 ceph-mon[25643]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:02 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:03.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:04.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:04 smithi098 ceph-mon[24773]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:04.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:04 smithi098 ceph-mon[29473]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:04.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:04 smithi153 ceph-mon[25643]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:06.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:05 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:05 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:05 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:06.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:05 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:48:06.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:05 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:48:06.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:05 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:48:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:06 smithi098 ceph-mon[24773]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:06 smithi098 ceph-mon[29473]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:07.094 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:06 smithi153 ceph-mon[25643]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:08 smithi153 ceph-mon[25643]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:08.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:08 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:08 smithi098 ceph-mon[24773]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:08 smithi098 ceph-mon[29473]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:09 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:10.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:09 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:09 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:10.814 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:48:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:48:10] "GET /metrics HTTP/1.1" 200 37901 "" "Prometheus/2.33.4" 2024-03-12T03:48:11.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:10 smithi098 ceph-mon[24773]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:11.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:10 smithi098 ceph-mon[29473]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:10 smithi153 ceph-mon[25643]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:12 smithi098 ceph-mon[29473]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:12 smithi098 ceph-mon[24773]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:12 smithi153 ceph-mon[25643]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:13.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:13 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:14 smithi153 ceph-mon[25643]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:15.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:14 smithi098 ceph-mon[24773]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:14 smithi098 ceph-mon[29473]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:16 smithi098 ceph-mon[29473]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:16 smithi098 ceph-mon[24773]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:16 smithi153 ceph-mon[25643]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:18.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:17 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:17 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:18.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:17 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:18 smithi153 ceph-mon[25643]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:19.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:18 smithi098 ceph-mon[24773]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:18 smithi098 ceph-mon[29473]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:19 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:20.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:19 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:20.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:19 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:20.830 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:48:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:48:20] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-03-12T03:48:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:20 smithi153 ceph-mon[25643]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:20 smithi098 ceph-mon[24773]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:20 smithi098 ceph-mon[29473]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:22 smithi153 ceph-mon[25643]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:23.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:22 smithi098 ceph-mon[24773]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:22 smithi098 ceph-mon[29473]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:23.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:24 smithi153 ceph-mon[25643]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:25.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:24 smithi098 ceph-mon[24773]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:24 smithi098 ceph-mon[29473]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:27.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:26 smithi098 ceph-mon[24773]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:27.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:26 smithi098 ceph-mon[29473]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:26 smithi153 ceph-mon[25643]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:28 smithi153 ceph-mon[25643]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:28.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:28 smithi098 ceph-mon[24773]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:28 smithi098 ceph-mon[29473]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:29 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:29 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:30.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:29 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:30.827 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:48:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:48:30] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-03-12T03:48:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:30 smithi153 ceph-mon[25643]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:31.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:30 smithi098 ceph-mon[24773]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:31.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:30 smithi098 ceph-mon[29473]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:32 smithi153 ceph-mon[25643]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:32 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:33.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:32 smithi098 ceph-mon[24773]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:32 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:33.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:32 smithi098 ceph-mon[29473]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:33.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:32 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:33.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:34 smithi153 ceph-mon[25643]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:35.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:34 smithi098 ceph-mon[24773]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:34 smithi098 ceph-mon[29473]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:37.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:36 smithi098 ceph-mon[24773]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:36 smithi098 ceph-mon[29473]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:36 smithi153 ceph-mon[25643]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:38.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:38 smithi153 ceph-mon[25643]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:38.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:39.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:38 smithi098 ceph-mon[24773]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:38 smithi098 ceph-mon[29473]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:40.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:39 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:40.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:39 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:40.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:39 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:40.825 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:48:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:48:40] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-03-12T03:48:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:40 smithi153 ceph-mon[25643]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:41.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:40 smithi098 ceph-mon[24773]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:40 smithi098 ceph-mon[29473]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:42 smithi153 ceph-mon[25643]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:43.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:42 smithi098 ceph-mon[24773]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:43.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:42 smithi098 ceph-mon[29473]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:43.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi098\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", domainname=\"(none)\", instance=\"smithi098\", job=\"node\", machine=\"x86_64\", nodename=\"smithi098\", release=\"5.14.0-427.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:44 smithi153 ceph-mon[25643]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:44 smithi098 ceph-mon[24773]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:44 smithi098 ceph-mon[29473]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:46 smithi098 ceph-mon[24773]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:46 smithi098 ceph-mon[29473]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:46 smithi153 ceph-mon[25643]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:47 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:48.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:47 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:48.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:47 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:48:48.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:48 smithi153 ceph-mon[25643]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:48.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:48:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:48:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"172.21.15.153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"ac0aaff8-e020-11ee-95c8-87774f69a715\", cluster_addr=\"172.21.15.98\", device_class=\"ssd\", hostname=\"smithi098\", instance=\"smithi153:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.98\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-12T03:48:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:48 smithi098 ceph-mon[24773]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:48 smithi098 ceph-mon[29473]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:49 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:50.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:49 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:50.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:49 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:48:50.839 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:48:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:48:50] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-03-12T03:48:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:50 smithi153 ceph-mon[25643]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:51.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:50 smithi098 ceph-mon[24773]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:50 smithi098 ceph-mon[29473]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:53.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:52 smithi153 ceph-mon[25643]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:53.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:52 smithi098 ceph-mon[24773]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:53.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:52 smithi098 ceph-mon[29473]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:54 smithi153 ceph-mon[25643]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:54 smithi098 ceph-mon[24773]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:54 smithi098 ceph-mon[29473]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:48:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:56 smithi098 ceph-mon[24773]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:56 smithi098 ceph-mon[29473]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:56 smithi153 ceph-mon[25643]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:58 smithi098 ceph-mon[24773]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:58 smithi098 ceph-mon[29473]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:48:59.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:58 smithi153 ceph-mon[25643]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:00.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:48:59 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:00.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:48:59 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:00.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:48:59 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:00.838 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:49:00] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-03-12T03:49:01.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:00 smithi153 ceph-mon[25643]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:01.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:00 smithi098 ceph-mon[24773]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:00 smithi098 ceph-mon[29473]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:03.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:02 smithi153 ceph-mon[25643]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:03.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:02 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:03.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:02 smithi098 ceph-mon[24773]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:02 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:03.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:02 smithi098 ceph-mon[29473]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:03.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:02 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:04.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:04 smithi153 ceph-mon[25643]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:05.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:04 smithi098 ceph-mon[24773]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:04 smithi098 ceph-mon[29473]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:07.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:06 smithi153 ceph-mon[25643]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:07.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:06 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:06 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:06 smithi153 ceph-mon[25643]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:49:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[24773]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[24773]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:49:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[29473]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:06 smithi098 ceph-mon[29473]: from='mgr.34760 ' entity='mgr.x' 2024-03-12T03:49:09.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:08 smithi098 ceph-mon[24773]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:08 smithi098 ceph-mon[29473]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:08 smithi153 ceph-mon[25643]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-12T03:49:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:09 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:10.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:09 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:10.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:09 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:10.835 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:49:10] "GET /metrics HTTP/1.1" 200 37893 "" "Prometheus/2.33.4" 2024-03-12T03:49:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:10 smithi153 ceph-mon[25643]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:10 smithi098 ceph-mon[24773]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:11.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:10 smithi098 ceph-mon[29473]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:12 smithi153 ceph-mon[25643]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:13.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:12 smithi098 ceph-mon[24773]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:13.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:12 smithi098 ceph-mon[29473]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:14 smithi153 ceph-mon[25643]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:15.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:14 smithi098 ceph-mon[24773]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:14 smithi098 ceph-mon[29473]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:16 smithi153 ceph-mon[25643]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:16 smithi098 ceph-mon[24773]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:16 smithi098 ceph-mon[29473]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:18.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:17 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:18.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:17 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:17 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:19.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:18 smithi098 ceph-mon[24773]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:18 smithi098 ceph-mon[29473]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:19.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:18 smithi153 ceph-mon[25643]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:19 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:20.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:19 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:20.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:19 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:20.847 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:49:20] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-03-12T03:49:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:20 smithi153 ceph-mon[25643]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:20 smithi098 ceph-mon[24773]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:20 smithi098 ceph-mon[29473]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:22 smithi153 ceph-mon[25643]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:23.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:22 smithi098 ceph-mon[24773]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:22 smithi098 ceph-mon[29473]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:24 smithi153 ceph-mon[25643]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:24 smithi098 ceph-mon[24773]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:24 smithi098 ceph-mon[29473]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:26 smithi153 ceph-mon[25643]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:27.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:26 smithi098 ceph-mon[24773]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:27.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:26 smithi098 ceph-mon[29473]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:28.970 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T03:49:29.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:28 smithi153 ceph-mon[25643]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:29.281 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:28 smithi098 ceph-mon[24773]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:29.282 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:28 smithi098 ceph-mon[29473]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:29.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:29 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:29.946 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:29 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:29 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:30.841 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:49:30] "GET /metrics HTTP/1.1" 200 37898 "" "Prometheus/2.33.4" 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (5m) 86s ago 13m 23.9M - 0.23.0 ba2b418f427c ae9380406106 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (5m) 2m ago 13m 49.5M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (4m) 86s ago 12m 53.1M - 3.5 e1d6a67b021e bd06cc3b03b7 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283 running (9m) 2m ago 18m 508M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 2e7f2996e113 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (2m) 86s ago 19m 448M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (19m) 86s ago 19m 56.1M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (18m) 2m ago 18m 47.9M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (18m) 86s ago 18m 45.8M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (6m) 86s ago 14m 17.0M - 1.3.1 1dbe0e931976 ff35fa2e462e 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (5m) 2m ago 14m 17.4M - 1.3.1 1dbe0e931976 78fb1d60e454 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (17m) 86s ago 17m 54.8M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:49:30.908 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (17m) 86s ago 17m 56.7M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (17m) 86s ago 17m 49.8M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (16m) 86s ago 16m 52.4M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (16m) 2m ago 16m 55.9M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (15m) 2m ago 15m 53.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (15m) 2m ago 15m 50.1M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (15m) 2m ago 15m 53.0M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (5m) 2m ago 14m 74.5M - 2.33.4 514e6a882f6e 8b54432f2054 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (12m) 86s ago 12m 73.6M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:49:30.909 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (12m) 2m ago 12m 75.7M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:49:31.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:30 smithi098 ceph-mon[24773]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:31.186 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:30 smithi098 ceph-mon[29473]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:30 smithi153 ceph-mon[25643]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:31.434 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions' 2024-03-12T03:49:32.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:31 smithi098 ceph-mon[24773]: from='client.25126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:49:32.040 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:31 smithi098 ceph-mon[29473]: from='client.25126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:49:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:31 smithi153 ceph-mon[25643]: from='client.25126 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:49:33.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:32 smithi098 ceph-mon[24773]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:33.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:32 smithi098 ceph-mon[24773]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:32 smithi098 ceph-mon[29473]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:32 smithi098 ceph-mon[29473]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:32 smithi153 ceph-mon[25643]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:32 smithi153 ceph-mon[25643]: from='mgr.34760 172.21.15.153:0/2429222147' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:33.268 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:49:33.269 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:49:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:49:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:49:33.270 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:49:33.270 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:49:33.951 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:33 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1476629515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:49:33.951 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:33 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1476629515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:49:33.973 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T03:49:34.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:33 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1476629515' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:49:35.150 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:34 smithi098 ceph-mon[24773]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:35.150 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:34 smithi098 ceph-mon[29473]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:34 smithi153 ceph-mon[25643]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:35.884 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T03:49:36.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph -s' 2024-03-12T03:49:37.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:36 smithi098 ceph-mon[24773]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:36 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2066811449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:49:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:36 smithi098 ceph-mon[29473]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:36 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2066811449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:49:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:36 smithi153 ceph-mon[25643]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:37.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:36 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2066811449' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: cluster: 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: id: ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: health: HEALTH_OK 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: services: 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: mgr: x(active, since 6m), standbys: y 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-03-12T03:49:38.506 INFO:teuthology.orchestra.run.smithi098.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: data: 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: pools: 6 pools, 161 pgs 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: objects: 241 objects, 585 KiB 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: usage: 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: pgs: 161 active+clean 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: io: 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:49:38.507 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:49:39.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:38 smithi098 ceph-mon[24773]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:39.180 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:38 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3215020253' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:49:39.180 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:38 smithi098 ceph-mon[29473]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:39.180 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:38 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3215020253' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:49:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:38 smithi153 ceph-mon[25643]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:49:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:38 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3215020253' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:49:39.203 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph mgr fail' 2024-03-12T03:49:40.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:39 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:40.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:39 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:40.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:39 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:40.861 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:49:40] "GET /metrics HTTP/1.1" 200 37896 "" "Prometheus/2.33.4" 2024-03-12T03:49:41.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:40 smithi098 ceph-mon[24773]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:40 smithi098 ceph-mon[29473]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:40 smithi153 ceph-mon[25643]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:49:42.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:41 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3049921356' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:49:42.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:41 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:49:42.113 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:41 smithi153 ceph-mon[25643]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:49:42.113 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:41.867+0000 7f05d3318700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-12T03:49:42.113 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ignoring --setuser ceph since I am not root 2024-03-12T03:49:42.113 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ignoring --setgroup ceph since I am not root 2024-03-12T03:49:42.113 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:41 smithi153 ceph-mgr[45394]: -- 172.21.15.153:0/3555675686 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5624869a45a0 con 0x562486978800 2024-03-12T03:49:42.114 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3049921356' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:49:42.114 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:49:42.114 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-mon[24773]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:49:42.114 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3049921356' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:49:42.115 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:49:42.115 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-mon[29473]: osdmap e104: 8 total, 8 up, 8 in 2024-03-12T03:49:42.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:49:41] ENGINE Bus STOPPING 2024-03-12T03:49:42.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:49:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:49:42.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:49:41] ENGINE Bus STOPPED 2024-03-12T03:49:42.115 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:49:41] ENGINE Bus STARTING 2024-03-12T03:49:42.116 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:49:42] ENGINE Serving on http://:::9283 2024-03-12T03:49:42.116 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:49:42] ENGINE Bus STARTED 2024-03-12T03:49:42.521 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'sleep 180' 2024-03-12T03:49:42.868 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:42.466+0000 7fc5cf0d0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:49:42.868 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:42.564+0000 7fc5cf0d0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:49:43.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: mgrmap e27: y(active, starting, since 0.681475s) 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:49:43.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: Manager daemon y is now available 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:49:43.091 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: mgrmap e27: y(active, starting, since 0.681475s) 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:49:43.092 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: Manager daemon y is now available 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:49:43.093 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:42 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:49:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: mgrmap e27: y(active, starting, since 0.681475s) 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:49:43.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: Manager daemon y is now available 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:49:43.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:42 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:49:43.514 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:43.265+0000 7fc5cf0d0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:49:43.514 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:43.360+0000 7fc5cf0d0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:49:43.514 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:43.509+0000 7fc5cf0d0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:49:43.882 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:43.613+0000 7fc5cf0d0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:49:43.883 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:43.699+0000 7fc5cf0d0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:49:44.125 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:43 smithi098 ceph-mon[24773]: mgrmap e28: y(active, since 1.69027s) 2024-03-12T03:49:44.125 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:43 smithi098 ceph-mon[24773]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:44.125 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:43 smithi098 ceph-mon[29473]: mgrmap e28: y(active, since 1.69027s) 2024-03-12T03:49:44.125 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:43 smithi098 ceph-mon[29473]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:44.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-mon[25643]: mgrmap e28: y(active, since 1.69027s) 2024-03-12T03:49:44.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-mon[25643]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:44.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:43.980+0000 7fc5cf0d0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:49:44.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:44.092+0000 7fc5cf0d0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:49:44.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:44.189+0000 7fc5cf0d0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:49:44.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:44.303+0000 7fc5cf0d0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:49:44.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:44.492+0000 7fc5cf0d0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:49:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:44 smithi153 ceph-mon[25643]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:44 smithi153 ceph-mon[25643]: mgrmap e29: y(active, since 2s) 2024-03-12T03:49:45.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:44 smithi098 ceph-mon[24773]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:45.230 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:44 smithi098 ceph-mon[24773]: mgrmap e29: y(active, since 2s) 2024-03-12T03:49:45.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:44 smithi098 ceph-mon[29473]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:45.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:44 smithi098 ceph-mon[29473]: mgrmap e29: y(active, since 2s) 2024-03-12T03:49:45.563 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:45.496+0000 7fc5cf0d0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:49:45.888 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:45.630+0000 7fc5cf0d0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[24773]: [12/Mar/2024:03:49:44] ENGINE Bus STARTING 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[24773]: [12/Mar/2024:03:49:44] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[24773]: [12/Mar/2024:03:49:44] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[24773]: [12/Mar/2024:03:49:44] ENGINE Bus STARTED 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[29473]: [12/Mar/2024:03:49:44] ENGINE Bus STARTING 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[29473]: [12/Mar/2024:03:49:44] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[29473]: [12/Mar/2024:03:49:44] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:49:46.179 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:45 smithi098 ceph-mon[29473]: [12/Mar/2024:03:49:44] ENGINE Bus STARTED 2024-03-12T03:49:46.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:45.985+0000 7fc5cf0d0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:49:46.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:46.080+0000 7fc5cf0d0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:49:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:45 smithi153 ceph-mon[25643]: [12/Mar/2024:03:49:44] ENGINE Bus STARTING 2024-03-12T03:49:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:45 smithi153 ceph-mon[25643]: [12/Mar/2024:03:49:44] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:49:46.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:45 smithi153 ceph-mon[25643]: [12/Mar/2024:03:49:44] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:49:46.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:45 smithi153 ceph-mon[25643]: [12/Mar/2024:03:49:44] ENGINE Bus STARTED 2024-03-12T03:49:46.480 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:46.286+0000 7fc5cf0d0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:49:46.480 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:46.477+0000 7fc5cf0d0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:49:47.069 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:46.789+0000 7fc5cf0d0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:49:47.069 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:47.065+0000 7fc5cf0d0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:49:47.069 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-mon[25643]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:47.069 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-mon[25643]: mgrmap e30: y(active, since 4s) 2024-03-12T03:49:47.069 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:47.069 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:46 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:47.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[24773]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:47.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[24773]: mgrmap e30: y(active, since 4s) 2024-03-12T03:49:47.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:47.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[29473]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[29473]: mgrmap e30: y(active, since 4s) 2024-03-12T03:49:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:47.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:46 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:47.326 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:47.323+0000 7fc5cf0d0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:49:48.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:48.147+0000 7fc5cf0d0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:49:48.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:48.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:48.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-mon[25643]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:48.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:48.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:48.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:48.720+0000 7fc5cf0d0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:49:48.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:48.819+0000 7fc5cf0d0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:49:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[24773]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[29473]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:48 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:49.038+0000 7fc5cf0d0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:49:49.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:49:49] ENGINE Bus STARTING 2024-03-12T03:49:49.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: CherryPy Checker: 2024-03-12T03:49:49.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: The Application mounted at '' has an empty config. 2024-03-12T03:49:49.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:49:49.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:49:49] ENGINE Serving on http://:::9283 2024-03-12T03:49:49.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:49:49] ENGINE Bus STARTED 2024-03-12T03:49:49.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:49.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: Standby manager daemon x started 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:49.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:49 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: Standby manager daemon x started 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='client.25090 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: Standby manager daemon x started 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:50.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:49 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:49:50.914 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:49:50.914 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:49:50.914 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:49:50.914 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:49:50.915 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:49:50.916 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:49:50.917 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:50 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:50.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:49:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: mgrmap e31: y(active, since 8s), standbys: x 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:50.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:49:50.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:49:50.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:50 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:49:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:51 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:49:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:51 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:49:51.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:51 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:49:51.948 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:51 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:49:51.948 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:51 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:49:51.948 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:51 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:49:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:52 smithi153 ceph-mon[25643]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:49:53.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:52 smithi098 ceph-mon[24773]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:49:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:52 smithi098 ceph-mon[29473]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:49:53.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:49:53.303+0000 7f293d322700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:49:53.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:53.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:53.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:53.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:53.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:53.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:53.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:53.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return future.result(timeout) 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.__get_result() 2024-03-12T03:49:53.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: raise self._exception 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: config_blobs=daemon_spec.final_config, 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:53.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:53.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:49:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:53.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:53.614Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:49:53.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:53.618Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:49:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:53.619Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:49:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:53.619Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:49:53.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:53.619Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:54 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:54.943 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:49:54.944 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:49:54.945 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:49:54.946 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:54 smithi153 ceph-mon[25643]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:54 smithi153 ceph-mon[25643]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:49:54.947 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:54 smithi153 ceph-mon[25643]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:49:55.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:49:55.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:55.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[24773]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[24773]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[24773]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:49:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:49:55.079 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:49:55.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:49:55.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:49:55.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[29473]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[29473]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:49:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:54 smithi098 ceph-mon[29473]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:49:56.936 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 systemd[1]: Stopping Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 bash[54523]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-12T03:49:56.937 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.862Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:49:56.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.863Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:49:56.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.863Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-12T03:49:56.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[49138]: ts=2024-03-12T03:49:56.863Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-12T03:49:56.938 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:56 smithi153 podman[54536]: 2024-03-12 03:49:56.872694504 +0000 UTC m=+0.117972375 container died 8b54432f20549a03a39fc8d1c7251e9692f81e098716fe68c780db8bf65270f0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:49:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:56 smithi153 ceph-mon[25643]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:49:57.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:56 smithi098 ceph-mon[24773]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:49:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:56 smithi098 ceph-mon[29473]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:49:57.600 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 podman[54536]: 2024-03-12 03:49:57.195126922 +0000 UTC m=+0.440404793 container cleanup 8b54432f20549a03a39fc8d1c7251e9692f81e098716fe68c780db8bf65270f0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:49:57.600 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 bash[54536]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:49:57.600 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 bash[54563]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:49:57.600 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 podman[54550]: 2024-03-12 03:49:57.371320368 +0000 UTC m=+0.500325589 container remove 8b54432f20549a03a39fc8d1c7251e9692f81e098716fe68c780db8bf65270f0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:49:57.600 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:49:57.600 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 systemd[1]: Stopped Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:49:57.601 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.943s CPU time. 2024-03-12T03:49:57.601 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 systemd[1]: Starting Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:49:57.901 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 podman[54637]: 2024-03-12 03:49:57.50281078 +0000 UTC m=+0.019627451 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-12T03:49:57.901 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 podman[54637]: 2024-03-12 03:49:57.639623526 +0000 UTC m=+0.156440194 container create ca984b8d91f4aa923323a91fa53ec43ba33c0d07e9ab0344077d334d4b45b4e0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:49:58.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 podman[54637]: 2024-03-12 03:49:57.899409323 +0000 UTC m=+0.416225997 container init ca984b8d91f4aa923323a91fa53ec43ba33c0d07e9ab0344077d334d4b45b4e0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 podman[54637]: 2024-03-12 03:49:57.90211499 +0000 UTC m=+0.418931665 container start ca984b8d91f4aa923323a91fa53ec43ba33c0d07e9ab0344077d334d4b45b4e0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.924Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.924Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.924Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.924Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi153 (none))" 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.924Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.924Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.926Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:49:58.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.926Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-12T03:49:58.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.928Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-12T03:49:58.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.930Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:49:58.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.930Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.579µs 2024-03-12T03:49:58.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:57.930Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:49:58.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 bash[54637]: ca984b8d91f4aa923323a91fa53ec43ba33c0d07e9ab0344077d334d4b45b4e0 2024-03-12T03:49:58.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:57 smithi153 systemd[1]: Started Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:49:58.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.141Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-12T03:49:58.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:57 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:57 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.364Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.568Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.568Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.568Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=33.891µs wal_replay_duration=638.207515ms total_replay_duration=638.258528ms 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.573Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.573Z caller=main.go:947 level=info msg="TSDB started" 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.573Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:49:58.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.594Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.745705ms db_storage=787ns remote_storage=1.239µs web_handler=781ns query_engine=710ns scrape=380.816µs scrape_sd=110.714µs notify=18.109µs notify_sd=22.735µs rules=20.870207ms 2024-03-12T03:49:58.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:49:58.594Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-12T03:49:59.130 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-mon[25643]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:49:59.130 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:59.130 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:59.130 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:58 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:49:59.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[24773]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:49:59.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:59.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:59.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:49:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[29473]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:49:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:49:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:58 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:50:00.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:49:59 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:50:00.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:49:59 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:50:00.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:49:59 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:50:01.265 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:00 smithi153 ceph-mon[25643]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:50:01.265 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:00 smithi153 ceph-mon[25643]: overall HEALTH_OK 2024-03-12T03:50:01.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:00 smithi098 ceph-mon[24773]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:50:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:00 smithi098 ceph-mon[24773]: overall HEALTH_OK 2024-03-12T03:50:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:00 smithi098 ceph-mon[29473]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:50:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:00 smithi098 ceph-mon[29473]: overall HEALTH_OK 2024-03-12T03:50:03.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:50:03.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:03.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:03.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.189 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:50:03.190 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:02 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:03.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:50:03.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:02 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:04.277 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:03 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:50:04.278 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:03 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:50:04.278 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:03 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:50:04.278 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:03 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:50:04.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:03 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:50:04.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:03 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:50:05.234 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:04 smithi098 ceph-mon[24773]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:50:05.234 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:04 smithi098 ceph-mon[29473]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:50:05.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:04 smithi153 ceph-mon[25643]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-03-12T03:50:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[24773]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:50:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:50:07.254 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:50:07.254 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[29473]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:50:07.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:07.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:07.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:50:07.255 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:06 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:50:07.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:06 smithi153 ceph-mon[25643]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:50:07.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:06 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:07.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:06 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:07.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:06 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:50:07.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:06 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:50:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:07 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:50:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:07 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:50:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:07 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2961165571' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:50:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:07 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:50:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:07 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:50:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:07 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2961165571' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:50:08.344 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:07 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:50:08.344 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:07 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:50:08.344 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:07 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2961165571' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:50:09.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:08 smithi098 ceph-mon[24773]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:50:09.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:08 smithi098 ceph-mon[29473]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:50:09.355 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:08 smithi153 ceph-mon[25643]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:50:10.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:50:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:50:10] "GET /metrics HTTP/1.1" 200 37910 "" "Prometheus/2.33.4" 2024-03-12T03:50:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:10 smithi098 ceph-mon[24773]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-12T03:50:11.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:10 smithi098 ceph-mon[29473]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-12T03:50:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:10 smithi153 ceph-mon[25643]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-12T03:50:12.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:12 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:12 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:12 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:12.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:12 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:12.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:12 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:12 smithi153 ceph-mon[25643]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-12T03:50:12.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:12 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[24773]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-12T03:50:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:12.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:50:12.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:50:12.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:50:12.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[29473]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 85 B/s rd, 0 op/s 2024-03-12T03:50:12.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:12 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:14 smithi153 ceph-mon[25643]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:15.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:14 smithi098 ceph-mon[24773]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:14 smithi098 ceph-mon[29473]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:17.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:16 smithi098 ceph-mon[24773]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:16 smithi098 ceph-mon[29473]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:16 smithi153 ceph-mon[25643]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:17 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:17 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:18.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:17 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:19.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:18 smithi098 ceph-mon[24773]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:18 smithi098 ceph-mon[29473]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:19.397 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:18 smithi153 ceph-mon[25643]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:50:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:50:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:50:20] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-03-12T03:50:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:20 smithi098 ceph-mon[24773]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:50:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:20 smithi098 ceph-mon[29473]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:50:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:20 smithi153 ceph-mon[25643]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:50:23.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:22 smithi098 ceph-mon[24773]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:22 smithi098 ceph-mon[29473]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:22 smithi153 ceph-mon[25643]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:24 smithi153 ceph-mon[25643]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:24 smithi098 ceph-mon[24773]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:24 smithi098 ceph-mon[29473]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:27.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:26 smithi098 ceph-mon[24773]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:27.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:26 smithi098 ceph-mon[29473]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:26 smithi153 ceph-mon[25643]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:28.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:27 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:27 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:28.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:27 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:28.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:27 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:27 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:27 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:29.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:28 smithi098 ceph-mon[24773]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:29.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:28 smithi098 ceph-mon[29473]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:29.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:28 smithi153 ceph-mon[25643]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:30.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:50:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:50:30] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-03-12T03:50:31.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:30 smithi098 ceph-mon[24773]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:31.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:30 smithi098 ceph-mon[29473]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:31.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:30 smithi153 ceph-mon[25643]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:33.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:32 smithi098 ceph-mon[24773]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:33.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:32 smithi098 ceph-mon[29473]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:32 smithi153 ceph-mon[25643]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:34 smithi153 ceph-mon[25643]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:34 smithi098 ceph-mon[24773]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:34 smithi098 ceph-mon[29473]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:37.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:36 smithi098 ceph-mon[24773]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:37.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:36 smithi098 ceph-mon[29473]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:37.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:36 smithi153 ceph-mon[25643]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:37 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:37 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:38.344 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:37 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:38 smithi098 ceph-mon[24773]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:38 smithi098 ceph-mon[29473]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:39.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:38 smithi153 ceph-mon[25643]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:40.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:50:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:50:40] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-03-12T03:50:41.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:40 smithi098 ceph-mon[24773]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:40 smithi098 ceph-mon[29473]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:41.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:40 smithi153 ceph-mon[25643]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:43.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:42 smithi098 ceph-mon[24773]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:43.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:42 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:43.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:42 smithi098 ceph-mon[29473]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:43.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:42 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:42 smithi153 ceph-mon[25643]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:42 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:44 smithi153 ceph-mon[25643]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:44 smithi098 ceph-mon[24773]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:44 smithi098 ceph-mon[29473]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:47.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:46 smithi098 ceph-mon[24773]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:46 smithi098 ceph-mon[29473]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:47.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:46 smithi153 ceph-mon[25643]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:47 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:48.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:47 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:47 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:49.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:48 smithi098 ceph-mon[24773]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:48 smithi098 ceph-mon[29473]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:48 smithi153 ceph-mon[25643]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:50.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:50:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:50:50] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-03-12T03:50:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:50 smithi098 ceph-mon[29473]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:51.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:50 smithi098 ceph-mon[24773]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:51.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:50 smithi153 ceph-mon[25643]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:53.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:52 smithi098 ceph-mon[29473]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:53.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:52 smithi098 ceph-mon[24773]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:53.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:52 smithi153 ceph-mon[25643]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:54 smithi153 ceph-mon[25643]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:54 smithi098 ceph-mon[29473]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:54 smithi098 ceph-mon[24773]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:57.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:56 smithi098 ceph-mon[24773]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:56 smithi098 ceph-mon[29473]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:57.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:56 smithi153 ceph-mon[25643]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:50:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:57 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:57 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:57 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:57 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:57 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:50:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:57 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:50:59.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:50:58 smithi098 ceph-mon[24773]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:50:58 smithi098 ceph-mon[29473]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:50:59.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:50:58 smithi153 ceph-mon[25643]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:51:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:51:00] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-03-12T03:51:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:00 smithi098 ceph-mon[24773]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:00 smithi098 ceph-mon[29473]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:00 smithi153 ceph-mon[25643]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:03.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:02 smithi098 ceph-mon[24773]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:03.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:02 smithi098 ceph-mon[29473]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:03.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:02 smithi153 ceph-mon[25643]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:04.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:04 smithi153 ceph-mon[25643]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:05.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:04 smithi098 ceph-mon[24773]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:04 smithi098 ceph-mon[29473]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:06 smithi098 ceph-mon[24773]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:06 smithi098 ceph-mon[29473]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:07.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:06 smithi153 ceph-mon[25643]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:07 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:07 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:08.344 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:07 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:09.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:08 smithi098 ceph-mon[24773]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:09.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:08 smithi098 ceph-mon[29473]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:09.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:08 smithi153 ceph-mon[25643]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:10.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:51:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:51:10] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-03-12T03:51:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:10 smithi098 ceph-mon[24773]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:11.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:10 smithi098 ceph-mon[29473]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:10 smithi153 ceph-mon[25643]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[24773]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[29473]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:51:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:51:13.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:12 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:12 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:12 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:12 smithi153 ceph-mon[25643]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:12 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:12 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:51:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:12 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:51:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:12 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:51:14.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:14 smithi153 ceph-mon[25643]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:15.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:14 smithi098 ceph-mon[24773]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:14 smithi098 ceph-mon[29473]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:16 smithi098 ceph-mon[24773]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:16 smithi098 ceph-mon[29473]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:17.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:16 smithi153 ceph-mon[25643]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:17 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:17 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:18.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:17 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:19.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:18 smithi098 ceph-mon[24773]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:18 smithi098 ceph-mon[29473]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:19.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:18 smithi153 ceph-mon[25643]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:51:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:51:20] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-03-12T03:51:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:20 smithi098 ceph-mon[24773]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:20 smithi098 ceph-mon[29473]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:20 smithi153 ceph-mon[25643]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:23.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:22 smithi098 ceph-mon[24773]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:22 smithi098 ceph-mon[29473]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:22 smithi153 ceph-mon[25643]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:24 smithi153 ceph-mon[25643]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:24 smithi098 ceph-mon[24773]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:24 smithi098 ceph-mon[29473]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:26 smithi098 ceph-mon[24773]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:27.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:26 smithi098 ceph-mon[29473]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:26 smithi153 ceph-mon[25643]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:27 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:28.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:27 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:28.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:27 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:28.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:27 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:27 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:27 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:29.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:28 smithi098 ceph-mon[24773]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:29.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:28 smithi098 ceph-mon[29473]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:29.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:28 smithi153 ceph-mon[25643]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:51:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:51:30] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-03-12T03:51:31.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:30 smithi098 ceph-mon[24773]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:31.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:30 smithi098 ceph-mon[29473]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:31.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:30 smithi153 ceph-mon[25643]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:33.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:32 smithi098 ceph-mon[24773]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:33.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:32 smithi098 ceph-mon[29473]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:32 smithi153 ceph-mon[25643]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:34 smithi153 ceph-mon[25643]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:34 smithi098 ceph-mon[24773]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:34 smithi098 ceph-mon[29473]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:37.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:36 smithi098 ceph-mon[24773]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:37.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:36 smithi098 ceph-mon[29473]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:37.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:36 smithi153 ceph-mon[25643]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:38.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:37 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:37 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:38.344 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:37 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:38 smithi098 ceph-mon[24773]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:38 smithi098 ceph-mon[29473]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:38 smithi153 ceph-mon[25643]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:40.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:51:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:51:40] "GET /metrics HTTP/1.1" 200 37894 "" "Prometheus/2.33.4" 2024-03-12T03:51:41.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:40 smithi098 ceph-mon[24773]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:40 smithi098 ceph-mon[29473]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:41.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:40 smithi153 ceph-mon[25643]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:42.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:41 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:42.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:41 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:42.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:41 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:43.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:42 smithi098 ceph-mon[24773]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:43.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:42 smithi098 ceph-mon[29473]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:42 smithi153 ceph-mon[25643]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:44 smithi153 ceph-mon[25643]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:44 smithi098 ceph-mon[24773]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:44 smithi098 ceph-mon[29473]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:47.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:46 smithi098 ceph-mon[24773]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:46 smithi098 ceph-mon[29473]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:47.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:46 smithi153 ceph-mon[25643]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:48.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:47 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:48.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:47 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:48.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:47 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:49.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:48 smithi098 ceph-mon[24773]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:48 smithi098 ceph-mon[29473]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:48 smithi153 ceph-mon[25643]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:50.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:51:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:51:50] "GET /metrics HTTP/1.1" 200 37902 "" "Prometheus/2.33.4" 2024-03-12T03:51:51.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:50 smithi098 ceph-mon[24773]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:50 smithi098 ceph-mon[29473]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:51.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:50 smithi153 ceph-mon[25643]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:53.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:52 smithi098 ceph-mon[24773]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:53.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:52 smithi098 ceph-mon[29473]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:53.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:52 smithi153 ceph-mon[25643]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:54 smithi153 ceph-mon[25643]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:54 smithi098 ceph-mon[24773]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:54 smithi098 ceph-mon[29473]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:57.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:56 smithi098 ceph-mon[24773]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:56 smithi098 ceph-mon[29473]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:57.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:56 smithi153 ceph-mon[25643]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:51:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:57 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:57 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:57 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:51:59.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:58 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:59.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:51:58 smithi098 ceph-mon[24773]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:58 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:51:58 smithi098 ceph-mon[29473]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:51:59.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:58 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:51:59.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:51:58 smithi153 ceph-mon[25643]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:52:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:52:00] "GET /metrics HTTP/1.1" 200 37916 "" "Prometheus/2.33.4" 2024-03-12T03:52:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:00 smithi098 ceph-mon[24773]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:00 smithi098 ceph-mon[29473]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:00 smithi153 ceph-mon[25643]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:03.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:02 smithi098 ceph-mon[24773]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:03.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:02 smithi098 ceph-mon[29473]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:03.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:02 smithi153 ceph-mon[25643]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:04.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:04 smithi153 ceph-mon[25643]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:04 smithi098 ceph-mon[24773]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:04 smithi098 ceph-mon[29473]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:06 smithi098 ceph-mon[24773]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:06 smithi098 ceph-mon[29473]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:07.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:06 smithi153 ceph-mon[25643]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:09.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:08 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:09.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:08 smithi153 ceph-mon[25643]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:09.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:08 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:09.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:08 smithi098 ceph-mon[24773]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:09.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:08 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:09.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:08 smithi098 ceph-mon[29473]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:10.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:52:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:52:10] "GET /metrics HTTP/1.1" 200 37916 "" "Prometheus/2.33.4" 2024-03-12T03:52:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:10 smithi098 ceph-mon[24773]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:11.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:10 smithi098 ceph-mon[29473]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:10 smithi153 ceph-mon[25643]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:12.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:11 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:12.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:11 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:12.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:11 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:13.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:12 smithi098 ceph-mon[24773]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:13.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:12 smithi098 ceph-mon[29473]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:13.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:12 smithi153 ceph-mon[25643]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[29473]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:52:14.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:14 smithi098 ceph-mon[24773]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:14 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:14 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:14.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:14 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:52:14.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:14 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:52:14.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:14 smithi153 ceph-mon[25643]: from='mgr.25120 ' entity='mgr.y' 2024-03-12T03:52:15.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:15 smithi098 ceph-mon[24773]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:15.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:15 smithi098 ceph-mon[29473]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:15.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:15 smithi153 ceph-mon[25643]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:17.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:17 smithi098 ceph-mon[24773]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:17 smithi098 ceph-mon[29473]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:17.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:17 smithi153 ceph-mon[25643]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:19.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:19 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:19.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:19 smithi098 ceph-mon[24773]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:19 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:19 smithi098 ceph-mon[29473]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:19.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:19 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:19.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:19 smithi153 ceph-mon[25643]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:20.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:52:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:52:20] "GET /metrics HTTP/1.1" 200 37916 "" "Prometheus/2.33.4" 2024-03-12T03:52:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:21 smithi098 ceph-mon[24773]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:21 smithi098 ceph-mon[29473]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:21 smithi153 ceph-mon[25643]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:22.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:22 smithi098 ceph-mon[24773]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:22.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:22 smithi098 ceph-mon[29473]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:22.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:22 smithi153 ceph-mon[25643]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:24 smithi153 ceph-mon[25643]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:24 smithi098 ceph-mon[24773]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:24 smithi098 ceph-mon[29473]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:26 smithi098 ceph-mon[24773]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:27.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:26 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:27.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:26 smithi098 ceph-mon[29473]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:27.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:26 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:26 smithi153 ceph-mon[25643]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:26 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:29.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:28 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:29.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:28 smithi098 ceph-mon[24773]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:29.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:28 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:29.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:28 smithi098 ceph-mon[29473]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:29.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:28 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:29.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:28 smithi153 ceph-mon[25643]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:52:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:52:30] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-03-12T03:52:31.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:30 smithi098 ceph-mon[24773]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:31.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:30 smithi098 ceph-mon[29473]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:31.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:30 smithi153 ceph-mon[25643]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:33.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:32 smithi098 ceph-mon[24773]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:33.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:32 smithi098 ceph-mon[29473]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:32 smithi153 ceph-mon[25643]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:34 smithi153 ceph-mon[25643]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:35.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:34 smithi098 ceph-mon[24773]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:34 smithi098 ceph-mon[29473]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:37.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:36 smithi098 ceph-mon[24773]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:37.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:36 smithi098 ceph-mon[29473]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:37.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:36 smithi153 ceph-mon[25643]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:39.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:38 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:39.245 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:38 smithi153 ceph-mon[25643]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:38 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:38 smithi098 ceph-mon[24773]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:38 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:38 smithi098 ceph-mon[29473]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:40.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:52:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:52:40] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-03-12T03:52:41.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:40 smithi098 ceph-mon[24773]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:40 smithi098 ceph-mon[29473]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:41.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:40 smithi153 ceph-mon[25643]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:42.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:41 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:42.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:41 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:42.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:41 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:43.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:42 smithi098 ceph-mon[24773]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:43.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:42 smithi098 ceph-mon[29473]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:42 smithi153 ceph-mon[25643]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:44.904 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:44 smithi098 ceph-mon[24773]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:44.904 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:44 smithi098 ceph-mon[29473]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:44.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:44 smithi153 ceph-mon[25643]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:45.314 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T03:52:47.281 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:52:47.281 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (9m) 2m ago 16m 23.9M - 0.23.0 ba2b418f427c ae9380406106 2024-03-12T03:52:47.281 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (8m) 2m ago 16m 57.4M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:52:47.281 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (2m) 2m ago 16m 43.8M - 3.5 e1d6a67b021e 2fd3a8b8d164 2024-03-12T03:52:47.281 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283 running (12m) 2m ago 21m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 2e7f2996e113 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (5m) 2m ago 23m 495M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (23m) 2m ago 23m 62.5M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (21m) 2m ago 21m 50.0M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (21m) 2m ago 21m 52.1M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (9m) 2m ago 17m 17.1M - 1.3.1 1dbe0e931976 ff35fa2e462e 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (8m) 2m ago 17m 17.4M - 1.3.1 1dbe0e931976 78fb1d60e454 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (21m) 2m ago 21m 55.1M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (20m) 2m ago 20m 57.1M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (20m) 2m ago 20m 50.0M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (19m) 2m ago 19m 52.7M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (19m) 2m ago 19m 56.5M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (19m) 2m ago 19m 53.8M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (18m) 2m ago 18m 50.8M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:52:47.282 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (18m) 2m ago 18m 53.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:52:47.283 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (2m) 2m ago 17m 41.9M - 2.33.4 514e6a882f6e ca984b8d91f4 2024-03-12T03:52:47.283 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (16m) 2m ago 16m 74.7M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:52:47.283 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (16m) 2m ago 16m 77.3M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:52:47.283 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:46 smithi098 ceph-mon[24773]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:47.283 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:46 smithi098 ceph-mon[29473]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:47.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:46 smithi153 ceph-mon[25643]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:47.920 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions' 2024-03-12T03:52:49.255 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:48 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:49.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:48 smithi098 ceph-mon[24773]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:52:49.256 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:48 smithi098 ceph-mon[24773]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:49.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:48 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:49.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:48 smithi098 ceph-mon[29473]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:52:49.256 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:48 smithi098 ceph-mon[29473]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:48 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:48 smithi153 ceph-mon[25643]: from='client.15309 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:52:49.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:48 smithi153 ceph-mon[25643]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:49.866 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:52:49.866 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:52:49.866 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:52:49.867 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:52:49.868 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:52:49.868 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:52:50.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:49 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3950220959' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:52:50.221 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:49 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3950220959' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:52:50.412 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph -s' 2024-03-12T03:52:50.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:49 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3950220959' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:52:50.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:52:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:52:50] "GET /metrics HTTP/1.1" 200 37915 "" "Prometheus/2.33.4" 2024-03-12T03:52:51.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:50 smithi098 ceph-mon[24773]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:50 smithi098 ceph-mon[29473]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:51.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:50 smithi153 ceph-mon[25643]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:52.383 INFO:teuthology.orchestra.run.smithi098.stdout: cluster: 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: id: ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: health: HEALTH_OK 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: services: 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: mgr: y(active, since 3m), standbys: x 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: data: 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: pools: 6 pools, 161 pgs 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: objects: 241 objects, 585 KiB 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: usage: 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:52:52.384 INFO:teuthology.orchestra.run.smithi098.stdout: pgs: 161 active+clean 2024-03-12T03:52:52.385 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:52:52.385 INFO:teuthology.orchestra.run.smithi098.stdout: io: 2024-03-12T03:52:52.385 INFO:teuthology.orchestra.run.smithi098.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-12T03:52:52.385 INFO:teuthology.orchestra.run.smithi098.stdout: 2024-03-12T03:52:53.021 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:52 smithi098 ceph-mon[24773]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:53.021 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:52 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2834367283' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:52:53.021 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:52 smithi098 ceph-mon[29473]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:53.021 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:52 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2834367283' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:52:53.047 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T03:52:53.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:52 smithi153 ceph-mon[25643]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:53.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:52 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2834367283' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-12T03:52:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:54 smithi098 ceph-mon[29473]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:54.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:54 smithi098 ceph-mon[24773]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:54 smithi153 ceph-mon[25643]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:55.050 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T03:52:55.709 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:55 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1568405823' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:52:55.709 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:55 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1568405823' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:52:55.711 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-12T03:52:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:55 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1568405823' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:52:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:56 smithi153 ceph-mon[25643]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:56.944 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:56 smithi098 ceph-mon[24773]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:56.945 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:56 smithi098 ceph-mon[29473]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:52:57.714 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:52:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:57 smithi153 ceph-mon[25643]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:58.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:57 smithi098 ceph-mon[24773]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:58.001 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:57 smithi098 ceph-mon[29473]: from='mgr.25120 172.21.15.98:0/1989453057' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:52:58.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph mgr fail' 2024-03-12T03:52:58.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:58 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:58.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:58 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1417951511' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:52:58.931 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:52:58 smithi098 ceph-mon[24773]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:58.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:58 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:58.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:58 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1417951511' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:52:58.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:52:58 smithi098 ceph-mon[29473]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:52:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:58 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:52:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:58 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1417951511' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:52:58.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:52:58 smithi153 ceph-mon[25643]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:53:00] "GET /metrics HTTP/1.1" 200 37915 "" "Prometheus/2.33.4" 2024-03-12T03:53:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[24773]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2222988028' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:53:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:53:01.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[24773]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:00.994+0000 7f29aac2d700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ignoring --setuser ceph since I am not root 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ignoring --setgroup ceph since I am not root 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:01 smithi098 ceph-mgr[63028]: -- 172.21.15.98:0/164684130 <== mon.1 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fa16db45a0 con 0x55fa16d88c00 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[29473]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2222988028' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:53:01.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:00 smithi098 ceph-mon[29473]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:53:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-mon[25643]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2222988028' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-mon[25643]: osdmap e105: 8 total, 8 up, 8 in 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:53:01] ENGINE Bus STOPPING 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:53:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:53:01] ENGINE Bus STOPPED 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:53:01] ENGINE Bus STARTING 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:53:01] ENGINE Serving on http://:::9283 2024-03-12T03:53:01.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:53:01] ENGINE Bus STARTED 2024-03-12T03:53:01.635 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'sleep 180' 2024-03-12T03:53:01.952 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:01.586+0000 7f5ffb163200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:53:01.953 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:01 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:01.690+0000 7f5ffb163200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: mgrmap e32: x(active, starting, since 0.666986s) 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:53:02.213 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: Manager daemon x is now available 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:53:02.214 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: [12/Mar/2024:03:53:01] ENGINE Bus STARTING 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: [12/Mar/2024:03:53:01] ENGINE Serving on http://172.21.15.153:8765 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: [12/Mar/2024:03:53:01] ENGINE Serving on https://172.21.15.153:7150 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[29473]: [12/Mar/2024:03:53:01] ENGINE Bus STARTED 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: mgrmap e32: x(active, starting, since 0.666986s) 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:53:02.215 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:53:02.216 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: Manager daemon x is now available 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: [12/Mar/2024:03:53:01] ENGINE Bus STARTING 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: [12/Mar/2024:03:53:01] ENGINE Serving on http://172.21.15.153:8765 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: [12/Mar/2024:03:53:01] ENGINE Serving on https://172.21.15.153:7150 2024-03-12T03:53:02.217 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-mon[24773]: [12/Mar/2024:03:53:01] ENGINE Bus STARTED 2024-03-12T03:53:02.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-12T03:53:02.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: mgrmap e32: x(active, starting, since 0.666986s) 2024-03-12T03:53:02.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:53:02.260 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: Manager daemon x is now available 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: [12/Mar/2024:03:53:01] ENGINE Bus STARTING 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: [12/Mar/2024:03:53:01] ENGINE Serving on http://172.21.15.153:8765 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: [12/Mar/2024:03:53:01] ENGINE Serving on https://172.21.15.153:7150 2024-03-12T03:53:02.261 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:02 smithi153 ceph-mon[25643]: [12/Mar/2024:03:53:01] ENGINE Bus STARTED 2024-03-12T03:53:02.484 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:02.387+0000 7f5ffb163200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:53:02.484 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:02.481+0000 7f5ffb163200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:53:02.812 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:02.628+0000 7f5ffb163200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:53:02.812 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:02.725+0000 7f5ffb163200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:53:02.812 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:02.809+0000 7f5ffb163200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:53:03.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-mon[24773]: mgrmap e33: x(active, since 1.68088s) 2024-03-12T03:53:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-mon[24773]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-mon[29473]: mgrmap e33: x(active, since 1.68088s) 2024-03-12T03:53:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-mon[29473]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:03.405 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:03.098+0000 7f5ffb163200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:53:03.405 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:03.213+0000 7f5ffb163200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:53:03.406 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:03.307+0000 7f5ffb163200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:53:03.406 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:03.402+0000 7f5ffb163200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:53:03.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:03 smithi153 ceph-mon[25643]: mgrmap e33: x(active, since 1.68088s) 2024-03-12T03:53:03.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:03 smithi153 ceph-mon[25643]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:03.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:03 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:03.595+0000 7f5ffb163200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:53:04.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:04 smithi098 ceph-mon[24773]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:04.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:04 smithi098 ceph-mon[24773]: mgrmap e34: x(active, since 2s) 2024-03-12T03:53:04.300 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:04 smithi098 ceph-mon[29473]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:04.300 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:04 smithi098 ceph-mon[29473]: mgrmap e34: x(active, since 2s) 2024-03-12T03:53:04.308 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:04 smithi153 ceph-mon[25643]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:04.308 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:04 smithi153 ceph-mon[25643]: mgrmap e34: x(active, since 2s) 2024-03-12T03:53:04.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:04.594+0000 7f5ffb163200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:53:04.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:04 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:04.733+0000 7f5ffb163200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:53:05.418 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:05 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:05.108+0000 7f5ffb163200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:53:05.419 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:05 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:05.203+0000 7f5ffb163200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:53:05.419 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:05 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:05.416+0000 7f5ffb163200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:53:05.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:05 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:05.612+0000 7f5ffb163200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:53:06.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[24773]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:06.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[24773]: mgrmap e35: x(active, since 4s) 2024-03-12T03:53:06.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:06.167 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:06.168 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:05 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:05.913+0000 7f5ffb163200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:53:06.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[29473]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:06.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[29473]: mgrmap e35: x(active, since 4s) 2024-03-12T03:53:06.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:06.168 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:06.432 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:06.182+0000 7f5ffb163200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:53:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:06 smithi153 ceph-mon[25643]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:06 smithi153 ceph-mon[25643]: mgrmap e35: x(active, since 4s) 2024-03-12T03:53:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:06 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:06 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:06.705 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:06 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:06.431+0000 7f5ffb163200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:53:07.565 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.565 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:07.324+0000 7f5ffb163200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:53:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:07 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:07 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:07 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:07.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:07 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:08.280 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:07 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:07.932+0000 7f5ffb163200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:53:08.280 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:08.041+0000 7f5ffb163200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:53:08.281 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:08.278+0000 7f5ffb163200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:53:08.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:53:08] ENGINE Bus STARTING 2024-03-12T03:53:08.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CherryPy Checker: 2024-03-12T03:53:08.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: The Application mounted at '' has an empty config. 2024-03-12T03:53:08.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:53:08.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:53:08] ENGINE Serving on http://:::9283 2024-03-12T03:53:08.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:53:08] ENGINE Bus STARTED 2024-03-12T03:53:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: Standby manager daemon y started 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:08 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: Standby manager daemon y started 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='client.15294 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: Standby manager daemon y started 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:53:09.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:08 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:53:09.844 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:53:09.845 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:09 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:09.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:53:09.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:53:09.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-12T03:53:09.952 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:53:09.953 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: mgrmap e36: x(active, since 8s), standbys: y 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.954 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.955 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:09.955 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:53:09.955 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:53:09.955 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:09 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:10.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:53:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:53:10.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:10 smithi153 ceph-mon[25643]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:53:10.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:10 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:53:10.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:10 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:53:10.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:10 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:53:10.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:10 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:53:10.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[24773]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:53:10.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:53:10.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:53:10.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:53:10.979 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:53:10.980 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[29473]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:53:10.980 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:53:10.980 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:53:10.980 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:53:10.980 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:10 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:53:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:12 smithi098 ceph-mon[24773]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:53:12.779 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:12 smithi098 ceph-mon[29473]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:53:12.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:12 smithi153 ceph-mon[25643]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:53:14.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:14 smithi153 ceph-mon[25643]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:53:14.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:14 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:14.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:14 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:14.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:14 smithi153 ceph-mon[25643]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:53:14.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:14 smithi153 ceph-mon[25643]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:53:14.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:14 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1632830720' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:53:14.619 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:14 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3841764647' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/199523389"}]: dispatch 2024-03-12T03:53:14.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[24773]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[24773]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[24773]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1632830720' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3841764647' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/199523389"}]: dispatch 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[29473]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:53:14.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:14.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:14.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[29473]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:53:14.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[29473]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:53:14.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1632830720' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:53:14.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:14 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3841764647' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/199523389"}]: dispatch 2024-03-12T03:53:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:15 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3841764647' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/199523389"}]': finished 2024-03-12T03:53:15.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:15 smithi153 ceph-mon[25643]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:53:15.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:15 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3765042220' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]: dispatch 2024-03-12T03:53:15.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:15 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]: dispatch 2024-03-12T03:53:16.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3841764647' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/199523389"}]': finished 2024-03-12T03:53:16.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[24773]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:53:16.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3765042220' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]: dispatch 2024-03-12T03:53:16.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]: dispatch 2024-03-12T03:53:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3841764647' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/199523389"}]': finished 2024-03-12T03:53:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[29473]: osdmap e106: 8 total, 8 up, 8 in 2024-03-12T03:53:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3765042220' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]: dispatch 2024-03-12T03:53:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:15 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]: dispatch 2024-03-12T03:53:16.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:16 smithi153 ceph-mon[25643]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:53:16.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:16 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]': finished 2024-03-12T03:53:16.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:16 smithi153 ceph-mon[25643]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:53:16.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:16 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/319264082' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]: dispatch 2024-03-12T03:53:16.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:16 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]: dispatch 2024-03-12T03:53:16.798 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:16 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:17.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[29473]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]': finished 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[29473]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/319264082' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]: dispatch 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]: dispatch 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[24773]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/1524623730"}]': finished 2024-03-12T03:53:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[24773]: osdmap e107: 8 total, 8 up, 8 in 2024-03-12T03:53:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/319264082' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]: dispatch 2024-03-12T03:53:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]: dispatch 2024-03-12T03:53:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:16 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:16 smithi153 systemd[1]: Stopping Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:16 smithi153 bash[57593]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.020Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.020Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.020Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.020Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.020Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.020Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-12T03:53:17.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.022Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:53:17.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.022Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-12T03:53:17.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.022Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:53:17.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.022Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:53:17.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.022Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-12T03:53:17.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[54651]: ts=2024-03-12T03:53:17.022Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-12T03:53:17.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 podman[57605]: 2024-03-12 03:53:17.030516181 +0000 UTC m=+0.142065281 container died ca984b8d91f4aa923323a91fa53ec43ba33c0d07e9ab0344077d334d4b45b4e0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:53:17.676 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 podman[57605]: 2024-03-12 03:53:17.399903202 +0000 UTC m=+0.511452348 container cleanup ca984b8d91f4aa923323a91fa53ec43ba33c0d07e9ab0344077d334d4b45b4e0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:53:17.676 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 bash[57605]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:53:17.676 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 bash[57631]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:53:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]': finished 2024-03-12T03:53:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-mon[25643]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:53:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/490831444' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]: dispatch 2024-03-12T03:53:17.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:17 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]: dispatch 2024-03-12T03:53:17.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 podman[57619]: 2024-03-12 03:53:17.676140699 +0000 UTC m=+0.645859892 container remove ca984b8d91f4aa923323a91fa53ec43ba33c0d07e9ab0344077d334d4b45b4e0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:53:17.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:53:17.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 systemd[1]: Stopped Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:53:17.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.392s CPU time. 2024-03-12T03:53:17.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 systemd[1]: Starting Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:53:17.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 podman[57704]: 2024-03-12 03:53:17.893676214 +0000 UTC m=+0.103449498 container create 4f896b5eb96e968702211cbf99e4312dee389b0b91b0d9efa92964f2760dedd5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:53:17.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:17 smithi153 podman[57704]: 2024-03-12 03:53:17.808960353 +0000 UTC m=+0.018733643 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-12T03:53:18.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]': finished 2024-03-12T03:53:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[24773]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:53:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/490831444' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]: dispatch 2024-03-12T03:53:18.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]: dispatch 2024-03-12T03:53:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3774653475"}]': finished 2024-03-12T03:53:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[29473]: osdmap e108: 8 total, 8 up, 8 in 2024-03-12T03:53:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/490831444' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]: dispatch 2024-03-12T03:53:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:17 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]: dispatch 2024-03-12T03:53:18.339 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 podman[57704]: 2024-03-12 03:53:18.095808575 +0000 UTC m=+0.305581868 container init 4f896b5eb96e968702211cbf99e4312dee389b0b91b0d9efa92964f2760dedd5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:53:18.339 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 podman[57704]: 2024-03-12 03:53:18.098426602 +0000 UTC m=+0.308199894 container start 4f896b5eb96e968702211cbf99e4312dee389b0b91b0d9efa92964f2760dedd5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:53:18.339 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.121Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-12T03:53:18.339 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.122Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-12T03:53:18.340 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.122Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-12T03:53:18.340 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.122Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi153 (none))" 2024-03-12T03:53:18.340 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.122Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:53:18.340 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.122Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:53:18.340 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.123Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:53:18.341 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.123Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-12T03:53:18.341 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.125Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-12T03:53:18.341 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.127Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:53:18.341 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.127Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.672µs 2024-03-12T03:53:18.341 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.127Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:53:18.341 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 bash[57704]: 4f896b5eb96e968702211cbf99e4312dee389b0b91b0d9efa92964f2760dedd5 2024-03-12T03:53:18.342 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 systemd[1]: Started Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:53:18.342 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.336Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-03-12T03:53:18.677 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.585Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-03-12T03:53:18.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]': finished 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/759726444' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1524623730"}]: dispatch 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:53:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.801Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.850Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.850Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.850Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=49.791µs wal_replay_duration=723.722048ms total_replay_duration=723.785339ms 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.855Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.855Z caller=main.go:947 level=info msg="TSDB started" 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.855Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.879Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.466681ms db_storage=926ns remote_storage=1.4µs web_handler=356ns query_engine=869ns scrape=380.381µs scrape_sd=119.949µs notify=18.51µs notify_sd=17.504µs rules=22.552346ms 2024-03-12T03:53:18.944 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:53:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:53:18.879Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]': finished 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/759726444' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1524623730"}]: dispatch 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:53:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3895679488"}]': finished 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: osdmap e109: 8 total, 8 up, 8 in 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/759726444' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1524623730"}]: dispatch 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:53:19.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:18 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:53:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:19 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:53:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:19 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:53:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:19 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:53:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:19 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/759726444' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1524623730"}]': finished 2024-03-12T03:53:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:19 smithi153 ceph-mon[25643]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:53:19.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:19 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1473597510' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/558348705"}]: dispatch 2024-03-12T03:53:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:53:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:53:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:53:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/759726444' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1524623730"}]': finished 2024-03-12T03:53:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[29473]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:53:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1473597510' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/558348705"}]: dispatch 2024-03-12T03:53:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:53:20.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:53:20.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:53:20.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/759726444' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/1524623730"}]': finished 2024-03-12T03:53:20.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[24773]: osdmap e110: 8 total, 8 up, 8 in 2024-03-12T03:53:20.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:19 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1473597510' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/558348705"}]: dispatch 2024-03-12T03:53:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:20 smithi098 ceph-mon[29473]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-12T03:53:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:20 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1473597510' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/558348705"}]': finished 2024-03-12T03:53:20.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:20 smithi098 ceph-mon[29473]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:53:20.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:20 smithi098 ceph-mon[24773]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-12T03:53:20.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:20 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1473597510' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/558348705"}]': finished 2024-03-12T03:53:20.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:20 smithi098 ceph-mon[24773]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:53:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:20 smithi153 ceph-mon[25643]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-12T03:53:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:20 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1473597510' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/558348705"}]': finished 2024-03-12T03:53:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:20 smithi153 ceph-mon[25643]: osdmap e111: 8 total, 8 up, 8 in 2024-03-12T03:53:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 1 op/s 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 1 op/s 2024-03-12T03:53:23.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:23.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:23.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:22 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 1 op/s 2024-03-12T03:53:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:53:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:53:23.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:22 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:53:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:24 smithi098 ceph-mon[24773]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:53:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:24 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:24 smithi098 ceph-mon[29473]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:53:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:24 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:24 smithi153 ceph-mon[25643]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T03:53:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:24 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:27.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:26 smithi098 ceph-mon[24773]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-03-12T03:53:27.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:26 smithi098 ceph-mon[29473]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-03-12T03:53:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:26 smithi153 ceph-mon[25643]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 697 B/s rd, 0 op/s 2024-03-12T03:53:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:28 smithi098 ceph-mon[24773]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-12T03:53:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:28 smithi098 ceph-mon[29473]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-12T03:53:29.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:28 smithi153 ceph-mon[25643]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2024-03-12T03:53:30.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:53:30] "GET /metrics HTTP/1.1" 200 37913 "" "Prometheus/2.33.4" 2024-03-12T03:53:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:30 smithi153 ceph-mon[25643]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:53:31.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:30 smithi098 ceph-mon[24773]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:53:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:30 smithi098 ceph-mon[29473]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T03:53:32.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:31 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:32.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:31 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:31 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:33.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:32 smithi098 ceph-mon[24773]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-12T03:53:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:32 smithi098 ceph-mon[29473]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-12T03:53:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:32 smithi153 ceph-mon[25643]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 904 B/s rd, 0 op/s 2024-03-12T03:53:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:34 smithi098 ceph-mon[29473]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:34 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:35.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:34 smithi098 ceph-mon[24773]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:35.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:34 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:34 smithi153 ceph-mon[25643]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:34 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:37.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:36 smithi098 ceph-mon[24773]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:36 smithi098 ceph-mon[29473]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:36 smithi153 ceph-mon[25643]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:39.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:38 smithi098 ceph-mon[24773]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:38 smithi098 ceph-mon[29473]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:38 smithi153 ceph-mon[25643]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:40 smithi153 ceph-mon[25643]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:40.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:53:40] "GET /metrics HTTP/1.1" 200 37912 "" "Prometheus/2.33.4" 2024-03-12T03:53:41.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:40 smithi098 ceph-mon[24773]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:40 smithi098 ceph-mon[29473]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:43.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:42 smithi098 ceph-mon[24773]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:42 smithi098 ceph-mon[29473]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:43.091 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:42 smithi153 ceph-mon[25643]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:44 smithi098 ceph-mon[24773]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:44 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:44 smithi098 ceph-mon[29473]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:44 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:44 smithi153 ceph-mon[25643]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:44 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:46 smithi098 ceph-mon[24773]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:46 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:46 smithi098 ceph-mon[29473]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:46 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:46 smithi153 ceph-mon[25643]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:46 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:53:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:48 smithi098 ceph-mon[24773]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:48 smithi098 ceph-mon[29473]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:48 smithi153 ceph-mon[25643]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:50.745 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:53:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:53:50] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-03-12T03:53:51.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:50 smithi098 ceph-mon[24773]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:50 smithi098 ceph-mon[29473]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:50 smithi153 ceph-mon[25643]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:52 smithi153 ceph-mon[25643]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:53.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:52 smithi098 ceph-mon[24773]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:52 smithi098 ceph-mon[29473]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:54 smithi098 ceph-mon[24773]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:54 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:54 smithi098 ceph-mon[29473]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:54 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:54 smithi153 ceph-mon[25643]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:54 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:53:57.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:56 smithi098 ceph-mon[24773]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:56 smithi098 ceph-mon[29473]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:56 smithi153 ceph-mon[25643]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:53:59.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:53:58 smithi098 ceph-mon[24773]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:59.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:53:58 smithi098 ceph-mon[29473]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:53:59.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:53:58 smithi153 ceph-mon[25643]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:00.764 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:54:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:54:00] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-03-12T03:54:01.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:00 smithi098 ceph-mon[24773]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:01.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:00 smithi098 ceph-mon[29473]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:01.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:00 smithi153 ceph-mon[25643]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:02.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:01 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:02.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:01 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:02.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:01 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:02 smithi098 ceph-mon[24773]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:02 smithi098 ceph-mon[29473]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:02 smithi153 ceph-mon[25643]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:04 smithi098 ceph-mon[24773]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:04 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:04 smithi098 ceph-mon[29473]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:04 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:04 smithi153 ceph-mon[25643]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:04 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:07.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:06 smithi098 ceph-mon[24773]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:06 smithi098 ceph-mon[29473]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:07.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:06 smithi153 ceph-mon[25643]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:09.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:08 smithi098 ceph-mon[24773]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:08 smithi098 ceph-mon[29473]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:08 smithi153 ceph-mon[25643]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:10.787 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:54:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:54:10] "GET /metrics HTTP/1.1" 200 37909 "" "Prometheus/2.33.4" 2024-03-12T03:54:11.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:10 smithi098 ceph-mon[24773]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:11.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:10 smithi098 ceph-mon[29473]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:10 smithi153 ceph-mon[25643]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:13.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:12 smithi098 ceph-mon[24773]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:13.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:12 smithi098 ceph-mon[29473]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:12 smithi153 ceph-mon[25643]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:15.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:14 smithi098 ceph-mon[24773]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:15.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:14 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:14 smithi098 ceph-mon[29473]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:14 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:14 smithi153 ceph-mon[25643]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:14 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:17.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:16 smithi098 ceph-mon[24773]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:16 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:16 smithi098 ceph-mon[29473]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:16 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:16 smithi153 ceph-mon[25643]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:16 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:19.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:18 smithi098 ceph-mon[24773]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:18 smithi098 ceph-mon[29473]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:19.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:18 smithi153 ceph-mon[25643]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:20.803 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:54:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:54:20] "GET /metrics HTTP/1.1" 200 37909 "" "Prometheus/2.33.4" 2024-03-12T03:54:21.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:20 smithi098 ceph-mon[24773]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:20 smithi098 ceph-mon[29473]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:20 smithi153 ceph-mon[25643]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:23.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:22 smithi098 ceph-mon[24773]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:22 smithi098 ceph-mon[29473]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:22 smithi153 ceph-mon[25643]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:24.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:23 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:54:24.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:23 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:54:24.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:23 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:54:24.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:23 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:54:24.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:23 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:54:24.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:23 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:54:24.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:23 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:54:24.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:23 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:54:24.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:23 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:54:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:24 smithi098 ceph-mon[29473]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:24 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:24 smithi098 ceph-mon[24773]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:24 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:24 smithi153 ceph-mon[25643]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:24 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:27.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:26 smithi098 ceph-mon[24773]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:27.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:26 smithi098 ceph-mon[29473]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:26 smithi153 ceph-mon[25643]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:29.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:28 smithi153 ceph-mon[25643]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:29.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:28 smithi098 ceph-mon[24773]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:29.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:28 smithi098 ceph-mon[29473]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:30.829 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:54:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:54:30] "GET /metrics HTTP/1.1" 200 37909 "" "Prometheus/2.33.4" 2024-03-12T03:54:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:30 smithi153 ceph-mon[25643]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:31.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:30 smithi098 ceph-mon[24773]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:31.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:30 smithi098 ceph-mon[29473]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:31 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:32.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:31 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:32.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:31 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:32 smithi153 ceph-mon[25643]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:33.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:32 smithi098 ceph-mon[24773]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:33.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:32 smithi098 ceph-mon[29473]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:34 smithi153 ceph-mon[25643]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:34 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:35.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:34 smithi098 ceph-mon[24773]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:35.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:34 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:34 smithi098 ceph-mon[29473]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:34 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:37.125 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:36 smithi098 ceph-mon[24773]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:37.126 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:36 smithi098 ceph-mon[29473]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:36 smithi153 ceph-mon[25643]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:38 smithi153 ceph-mon[25643]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:39.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:38 smithi098 ceph-mon[24773]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:38 smithi098 ceph-mon[29473]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:40.848 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:54:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:54:40] "GET /metrics HTTP/1.1" 200 37908 "" "Prometheus/2.33.4" 2024-03-12T03:54:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:40 smithi153 ceph-mon[25643]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:41.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:40 smithi098 ceph-mon[24773]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:40 smithi098 ceph-mon[29473]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:42 smithi153 ceph-mon[25643]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:43.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:42 smithi098 ceph-mon[29473]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:43.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:42 smithi098 ceph-mon[24773]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:44 smithi153 ceph-mon[25643]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:44 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:45.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:44 smithi098 ceph-mon[29473]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:45.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:44 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:45.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:44 smithi098 ceph-mon[24773]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:45.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:44 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:46 smithi153 ceph-mon[25643]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:46 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:47.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:46 smithi098 ceph-mon[24773]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:47.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:46 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:46 smithi098 ceph-mon[29473]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:46 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:54:49.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:48 smithi153 ceph-mon[25643]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:49.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:48 smithi098 ceph-mon[24773]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:48 smithi098 ceph-mon[29473]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:50.865 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:54:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:54:50] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-03-12T03:54:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:50 smithi153 ceph-mon[25643]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:51.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:50 smithi098 ceph-mon[24773]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:50 smithi098 ceph-mon[29473]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:53.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:52 smithi153 ceph-mon[25643]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:53.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:52 smithi098 ceph-mon[24773]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:53.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:52 smithi098 ceph-mon[29473]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:54 smithi153 ceph-mon[25643]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:54 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:55.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:54 smithi098 ceph-mon[24773]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:55.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:54 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:55.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:54 smithi098 ceph-mon[29473]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:55.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:54 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:54:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:56 smithi153 ceph-mon[25643]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:57.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:56 smithi098 ceph-mon[24773]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:56 smithi098 ceph-mon[29473]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:54:59.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:54:58 smithi153 ceph-mon[25643]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:59.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:54:58 smithi098 ceph-mon[24773]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:54:59.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:54:58 smithi098 ceph-mon[29473]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:00.888 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:55:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:55:00] "GET /metrics HTTP/1.1" 200 37906 "" "Prometheus/2.33.4" 2024-03-12T03:55:01.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:00 smithi153 ceph-mon[25643]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:01.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:00 smithi098 ceph-mon[24773]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:01.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:00 smithi098 ceph-mon[29473]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:02.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:01 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:02.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:01 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:02.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:01 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:02 smithi153 ceph-mon[25643]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:03.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:02 smithi098 ceph-mon[24773]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:03.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:02 smithi098 ceph-mon[29473]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:04 smithi153 ceph-mon[25643]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:04 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:05.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:04 smithi098 ceph-mon[24773]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:05.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:04 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:05.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:04 smithi098 ceph-mon[29473]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:05.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:04 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:07.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:06 smithi153 ceph-mon[25643]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:07.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:06 smithi098 ceph-mon[24773]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:06 smithi098 ceph-mon[29473]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:08 smithi153 ceph-mon[25643]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:09.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:08 smithi098 ceph-mon[24773]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:09.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:08 smithi098 ceph-mon[29473]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:10.907 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:55:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:55:10] "GET /metrics HTTP/1.1" 200 37904 "" "Prometheus/2.33.4" 2024-03-12T03:55:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:10 smithi153 ceph-mon[25643]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:10 smithi098 ceph-mon[24773]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:11.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:10 smithi098 ceph-mon[29473]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:12 smithi153 ceph-mon[25643]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:13.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:12 smithi098 ceph-mon[24773]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:13.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:12 smithi098 ceph-mon[29473]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:14 smithi153 ceph-mon[25643]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:14 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:15.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:14 smithi098 ceph-mon[24773]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:15.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:14 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:15.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:14 smithi098 ceph-mon[29473]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:15.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:14 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:16 smithi153 ceph-mon[25643]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:16 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:17.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:16 smithi098 ceph-mon[29473]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:16 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:16 smithi098 ceph-mon[24773]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:16 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:19.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:18 smithi153 ceph-mon[25643]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:19.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:18 smithi098 ceph-mon[24773]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:18 smithi098 ceph-mon[29473]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:20.922 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:55:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:55:20] "GET /metrics HTTP/1.1" 200 37917 "" "Prometheus/2.33.4" 2024-03-12T03:55:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:20 smithi153 ceph-mon[25643]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:20 smithi098 ceph-mon[24773]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:20 smithi098 ceph-mon[29473]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:22 smithi153 ceph-mon[25643]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:23.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:22 smithi098 ceph-mon[24773]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:22 smithi098 ceph-mon[29473]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:24 smithi153 ceph-mon[25643]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:24 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:24 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:55:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:24 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:55:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:24 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:55:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[24773]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:55:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:55:25.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:55:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[29473]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:25.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:55:25.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:55:25.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:24 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:55:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:26 smithi153 ceph-mon[25643]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:27.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:26 smithi098 ceph-mon[24773]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:27.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:26 smithi098 ceph-mon[29473]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:28 smithi153 ceph-mon[25643]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:29.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:28 smithi098 ceph-mon[24773]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:28 smithi098 ceph-mon[29473]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:30.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:30 smithi153 ceph-mon[25643]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:30.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:55:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:55:30] "GET /metrics HTTP/1.1" 200 37917 "" "Prometheus/2.33.4" 2024-03-12T03:55:31.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:30 smithi098 ceph-mon[24773]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:30 smithi098 ceph-mon[29473]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:32.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:31 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:32.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:31 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:31 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:33.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:32 smithi098 ceph-mon[24773]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:33.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:32 smithi098 ceph-mon[29473]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:33.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:32 smithi153 ceph-mon[25643]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:34 smithi098 ceph-mon[24773]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:34 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:34 smithi098 ceph-mon[29473]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:34 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:34 smithi153 ceph-mon[25643]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:34 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:37.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:36 smithi098 ceph-mon[24773]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:37.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:36 smithi098 ceph-mon[29473]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:36 smithi153 ceph-mon[25643]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:39.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:38 smithi098 ceph-mon[24773]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:39.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:38 smithi098 ceph-mon[29473]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:38 smithi153 ceph-mon[25643]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:40 smithi153 ceph-mon[25643]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:40.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:55:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:55:40] "GET /metrics HTTP/1.1" 200 37917 "" "Prometheus/2.33.4" 2024-03-12T03:55:41.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:40 smithi098 ceph-mon[24773]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:41.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:40 smithi098 ceph-mon[29473]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:43.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:42 smithi098 ceph-mon[24773]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:43.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:42 smithi098 ceph-mon[29473]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:42 smithi153 ceph-mon[25643]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:45.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:44 smithi098 ceph-mon[24773]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:45.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:44 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:44 smithi098 ceph-mon[29473]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:44 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:44 smithi153 ceph-mon[25643]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:45.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:44 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:46 smithi098 ceph-mon[24773]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:46 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:46 smithi098 ceph-mon[29473]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:46 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:46 smithi153 ceph-mon[25643]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:46 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:55:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:48 smithi098 ceph-mon[24773]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:48 smithi098 ceph-mon[29473]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:48 smithi153 ceph-mon[25643]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:50.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:50 smithi153 ceph-mon[25643]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:50.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:55:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:55:50] "GET /metrics HTTP/1.1" 200 37917 "" "Prometheus/2.33.4" 2024-03-12T03:55:51.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:50 smithi098 ceph-mon[24773]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:50 smithi098 ceph-mon[29473]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:53.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:52 smithi098 ceph-mon[24773]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:52 smithi098 ceph-mon[29473]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:53.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:52 smithi153 ceph-mon[25643]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:55.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:54 smithi098 ceph-mon[24773]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:55.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:54 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:54 smithi098 ceph-mon[29473]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:54 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:54 smithi153 ceph-mon[25643]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:54 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:55:57.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:56 smithi098 ceph-mon[24773]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:56 smithi098 ceph-mon[29473]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:57.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:56 smithi153 ceph-mon[25643]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:55:59.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:55:58 smithi098 ceph-mon[24773]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:55:58 smithi098 ceph-mon[29473]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:55:59.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:55:58 smithi153 ceph-mon[25643]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:00.742 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:56:00] "GET /metrics HTTP/1.1" 200 37917 "" "Prometheus/2.33.4" 2024-03-12T03:56:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:00 smithi098 ceph-mon[24773]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:01.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:00 smithi098 ceph-mon[29473]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:01.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:00 smithi153 ceph-mon[25643]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:02.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:01 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:02.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:01 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:02.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:01 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:03.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:02 smithi098 ceph-mon[24773]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:02 smithi098 ceph-mon[29473]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:02 smithi153 ceph-mon[25643]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:04.696 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T03:56:05.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:04 smithi098 ceph-mon[24773]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:05.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:04 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:05.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:04 smithi098 ceph-mon[29473]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:05.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:04 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:04 smithi153 ceph-mon[25643]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:04 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:06.599 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (12m) 2m ago 20m 22.7M - 0.23.0 ba2b418f427c ae9380406106 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (11m) 2m ago 19m 57.6M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (2m) 2m ago 19m 51.7M - 3.5 e1d6a67b021e 72e16f5fa11f 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283 running (15m) 2m ago 24m 492M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 2e7f2996e113 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (9m) 2m ago 26m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (26m) 2m ago 26m 65.5M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (25m) 2m ago 25m 53.1M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (25m) 2m ago 25m 54.5M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (12m) 2m ago 21m 17.1M - 1.3.1 1dbe0e931976 ff35fa2e462e 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (12m) 2m ago 20m 17.6M - 1.3.1 1dbe0e931976 78fb1d60e454 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (24m) 2m ago 24m 55.4M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:56:06.600 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (24m) 2m ago 24m 57.3M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (23m) 2m ago 23m 50.3M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (23m) 2m ago 23m 51.9M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (22m) 2m ago 22m 56.7M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (22m) 2m ago 22m 53.9M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (22m) 2m ago 22m 51.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (21m) 2m ago 21m 54.2M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (2m) 2m ago 21m 44.3M - 2.33.4 514e6a882f6e 4f896b5eb96e 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (19m) 2m ago 19m 75.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:56:06.601 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (19m) 2m ago 19m 78.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:56:06.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:06 smithi098 ceph-mon[24773]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:06.966 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:06 smithi098 ceph-mon[29473]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:06 smithi153 ceph-mon[25643]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:07.282 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ls' 2024-03-12T03:56:07.803 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:07 smithi098 ceph-mon[24773]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:07.804 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:07 smithi098 ceph-mon[29473]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:08.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:07 smithi153 ceph-mon[25643]: from='client.15432 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:08.969 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:08 smithi098 ceph-mon[24773]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:08.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:08 smithi098 ceph-mon[29473]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:08 smithi153 ceph-mon[25643]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-12T03:56:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager ?:9093,9094 1/1 2m ago 21m smithi098=a;count:1 2024-03-12T03:56:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:grafana ?:3000 1/1 2m ago 21m smithi153=a;count:1 2024-03-12T03:56:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo ?:5000 1/1 2m ago 19m count:1 2024-03-12T03:56:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:mgr 2/2 2m ago 24m smithi098=y;smithi153=x;count:2 2024-03-12T03:56:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:mon 3/3 2m ago 25m smithi098:172.21.15.98=a;smithi098:[v2:172.21.15.98:3301,v1:172.21.15.98:6790]=c;smithi153:172.21.15.153=b;count:3 2024-03-12T03:56:09.211 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter ?:9100 2/2 2m ago 21m smithi098=a;smithi153=b;count:2 2024-03-12T03:56:09.212 INFO:teuthology.orchestra.run.smithi098.stdout:osd 8 2m ago - 2024-03-12T03:56:09.212 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus ?:9095 1/1 2m ago 21m smithi153=a;count:1 2024-03-12T03:56:09.212 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo ?:8000 2/2 2m ago 19m count:2 2024-03-12T03:56:09.830 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions' 2024-03-12T03:56:10.765 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:10 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:56:10] "GET /metrics HTTP/1.1" 200 37917 "" "Prometheus/2.33.4" 2024-03-12T03:56:11.065 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:10 smithi098 ceph-mon[24773]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:11.065 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:10 smithi098 ceph-mon[24773]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:11.065 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:10 smithi098 ceph-mon[29473]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:11.065 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:10 smithi098 ceph-mon[29473]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:10 smithi153 ceph-mon[25643]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:10 smithi153 ceph-mon[25643]: from='client.15438 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:11.829 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:56:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:56:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:56:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:11.829 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:56:11.830 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:56:12.421 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-12T03:56:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:12 smithi098 ceph-mon[24773]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:12 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/891183561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:12 smithi098 ceph-mon[29473]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:12 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/891183561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:12 smithi153 ceph-mon[25643]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:12 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/891183561' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:15.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:14 smithi098 ceph-mon[24773]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:15.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:14 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:14 smithi098 ceph-mon[29473]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:15.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:14 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:14 smithi153 ceph-mon[25643]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:14 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:15.913 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:16.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:15 smithi098 ceph-mon[24773]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:16.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:15 smithi098 ceph-mon[29473]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:16.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:15 smithi153 ceph-mon[25643]: from='client.25351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:16.517 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T03:56:17.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[24773]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:17.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[24773]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[29473]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[29473]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:17.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:17.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:17.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:16 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:16 smithi153 ceph-mon[25643]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:56:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:16 smithi153 ceph-mon[25643]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:16 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:16 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:16 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:16 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:17.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:16 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:18.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:17 smithi153 ceph-mon[25643]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:18.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:17 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:18.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:17 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:18.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:17 smithi153 ceph-mon[25643]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-12T03:56:18.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:17 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-12T03:56:18.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:17 smithi153 ceph-mon[25643]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:56:18.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[24773]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[24773]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[24773]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[29473]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[29473]: from='mgr.25162 172.21.15.153:0/3929605493' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-12T03:56:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:17 smithi098 ceph-mon[29473]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:56:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:18.859Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:56:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:18.861Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:56:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:18.861Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:56:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:18.863Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:56:18.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:18.863Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.153:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.153:8765: connect: connection refused" 2024-03-12T03:56:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:18.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: Failing over to other MGR 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25162 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: mgrmap e37: y(active, starting, since 0.996166s) 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:56:18.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: Manager daemon y is now available 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:56:18.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:56:18.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:56:18.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:18.675+0000 7fc5c2ae7700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-12T03:56:18.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ignoring --setuser ceph since I am not root 2024-03-12T03:56:18.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ignoring --setgroup ceph since I am not root 2024-03-12T03:56:18.946 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:18 smithi153 ceph-mgr[45394]: -- 172.21.15.153:0/3959250298 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f46a6d05a0 con 0x55f46a6a4800 2024-03-12T03:56:18.963 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:56:18] ENGINE Bus STOPPING 2024-03-12T03:56:18.963 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:56:18] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:56:18] ENGINE Bus STOPPED 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:56:18] ENGINE Bus STARTING 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:56:18] ENGINE Serving on http://:::9283 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:56:18] ENGINE Bus STARTED 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:56:18.964 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: Failing over to other MGR 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25162 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: mgrmap e37: y(active, starting, since 0.996166s) 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:56:18.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: Manager daemon y is now available 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:56:18.966 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:56:18.967 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:56:18.967 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: Failing over to other MGR 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25162 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: mgrmap e37: y(active, starting, since 0.996166s) 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:56:18.968 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: Manager daemon y is now available 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:56:18.969 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:56:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:56:18.970 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:18 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:56:19.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:19.253+0000 7f2927a5d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:56:19.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:19.350+0000 7f2927a5d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:56:19.714 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:56:20.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:19 smithi098 ceph-mon[24773]: [12/Mar/2024:03:56:19] ENGINE Bus STARTING 2024-03-12T03:56:20.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:19 smithi098 ceph-mon[24773]: mgrmap e38: y(active, since 2s) 2024-03-12T03:56:20.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:19 smithi098 ceph-mon[29473]: [12/Mar/2024:03:56:19] ENGINE Bus STARTING 2024-03-12T03:56:20.045 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:19 smithi098 ceph-mon[29473]: mgrmap e38: y(active, since 2s) 2024-03-12T03:56:20.135 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:19 smithi153 ceph-mon[25643]: [12/Mar/2024:03:56:19] ENGINE Bus STARTING 2024-03-12T03:56:20.136 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:19 smithi153 ceph-mon[25643]: mgrmap e38: y(active, since 2s) 2024-03-12T03:56:20.136 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.040+0000 7f2927a5d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:56:20.136 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.133+0000 7f2927a5d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:56:20.372 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:56:20.372 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (12m) 2m ago 20m 22.7M - 0.23.0 ba2b418f427c ae9380406106 2024-03-12T03:56:20.372 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (12m) 2m ago 20m 57.6M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:56:20.372 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (3m) 2m ago 19m 51.7M - 3.5 e1d6a67b021e 72e16f5fa11f 2024-03-12T03:56:20.372 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283 running (15m) 2m ago 24m 492M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 2e7f2996e113 2024-03-12T03:56:20.372 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (9m) 2m ago 26m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (26m) 2m ago 26m 65.5M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (25m) 2m ago 25m 53.1M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (25m) 2m ago 25m 54.5M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (12m) 2m ago 21m 17.1M - 1.3.1 1dbe0e931976 ff35fa2e462e 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (12m) 2m ago 21m 17.6M - 1.3.1 1dbe0e931976 78fb1d60e454 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (24m) 2m ago 24m 55.4M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (24m) 2m ago 24m 57.3M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (23m) 2m ago 23m 50.3M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (23m) 2m ago 23m 51.9M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (23m) 2m ago 23m 56.7M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (22m) 2m ago 22m 53.9M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (22m) 2m ago 22m 51.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (22m) 2m ago 22m 54.2M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:56:20.373 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (3m) 2m ago 21m 44.3M - 2.33.4 514e6a882f6e 4f896b5eb96e 2024-03-12T03:56:20.374 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (19m) 2m ago 19m 75.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:56:20.374 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (19m) 2m ago 19m 78.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:56:20.392 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.279+0000 7f2927a5d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:56:20.392 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.390+0000 7f2927a5d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:56:20.682 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.474+0000 7f2927a5d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 1 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:56:20.779 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 1 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:56:20.780 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:56:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-mon[25643]: [12/Mar/2024:03:56:19] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:56:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-mon[25643]: [12/Mar/2024:03:56:19] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:56:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-mon[25643]: [12/Mar/2024:03:56:19] ENGINE Bus STARTED 2024-03-12T03:56:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-mon[25643]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-mon[25643]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:20.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-mon[25643]: from='client.25387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:20.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/866773249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:20.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.752+0000 7f2927a5d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:56:20.943 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.866+0000 7f2927a5d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:56:21.111 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:56:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T03:56:21.111 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [ 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr" 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "2/2 daemons upgraded", 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:56:21.112 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:56:21.290 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:20.955+0000 7f2927a5d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:56:21.291 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:21 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:21.078+0000 7f2927a5d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:56:21.291 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:21 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:21.289+0000 7f2927a5d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:56:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[24773]: [12/Mar/2024:03:56:19] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:56:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[24773]: [12/Mar/2024:03:56:19] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:56:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[24773]: [12/Mar/2024:03:56:19] ENGINE Bus STARTED 2024-03-12T03:56:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[24773]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[24773]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[24773]: from='client.25387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/866773249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[29473]: [12/Mar/2024:03:56:19] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[29473]: [12/Mar/2024:03:56:19] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[29473]: [12/Mar/2024:03:56:19] ENGINE Bus STARTED 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[29473]: from='client.25381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[29473]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[29473]: from='client.25387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:20 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/866773249' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:22.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:21 smithi153 ceph-mon[25643]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:22.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:21 smithi153 ceph-mon[25643]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:22.302 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:21 smithi098 ceph-mon[24773]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:22.302 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:21 smithi098 ceph-mon[24773]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:22.302 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:21 smithi098 ceph-mon[29473]: from='client.15492 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:22.302 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:21 smithi098 ceph-mon[29473]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:22.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:22.301+0000 7f2927a5d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:56:22.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:22.434+0000 7f2927a5d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:56:23.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:22 smithi153 ceph-mon[25643]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:23.092 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:22 smithi153 ceph-mon[25643]: mgrmap e39: y(active, since 4s) 2024-03-12T03:56:23.092 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:22.784+0000 7f2927a5d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:56:23.092 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:22.887+0000 7f2927a5d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:56:23.092 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:23.090+0000 7f2927a5d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:56:23.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:22 smithi098 ceph-mon[24773]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:23.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:22 smithi098 ceph-mon[24773]: mgrmap e39: y(active, since 4s) 2024-03-12T03:56:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:22 smithi098 ceph-mon[29473]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:22 smithi098 ceph-mon[29473]: mgrmap e39: y(active, since 4s) 2024-03-12T03:56:23.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:23.287+0000 7f2927a5d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:56:23.823 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=warn ts=2024-03-12T03:56:23.541Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T03:56:23.873 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:23.593+0000 7f2927a5d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:56:23.874 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:23.871+0000 7f2927a5d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:56:24.127 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:24.125+0000 7f2927a5d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:56:24.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-mon[25643]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:24.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.128 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:23 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[24773]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[29473]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:24.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:23 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:24.987+0000 7f2927a5d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:56:25.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-mon[25643]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:25.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:25.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:24 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:25.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[24773]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:25.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[29473]: from='client.15381 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:24 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:25.920 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:25.592+0000 7f2927a5d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:56:25.921 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:25.697+0000 7f2927a5d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:56:26.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:25.919+0000 7f2927a5d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:56:26.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:56:25] ENGINE Bus STARTING 2024-03-12T03:56:26.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: CherryPy Checker: 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: The Application mounted at '' has an empty config. 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:56:26] ENGINE Serving on http://:::9283 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: [12/Mar/2024:03:56:26] ENGINE Bus STARTED 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:56:26.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:26.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:25 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.236 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:56:26.237 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:26.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:25 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:56:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:56:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: Standby manager daemon x started 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:56:27.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:26 smithi153 ceph-mon[25643]: mgrmap e40: y(active, since 8s), standbys: x 2024-03-12T03:56:27.236 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:56:26.945Z caller=notify.go:735 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=5 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: Standby manager daemon x started 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:56:27.237 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[24773]: mgrmap e40: y(active, since 8s), standbys: x 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: Standby manager daemon x started 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/2155660092' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:56:27.238 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:26 smithi098 ceph-mon[29473]: mgrmap e40: y(active, since 8s), standbys: x 2024-03-12T03:56:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:27 smithi153 ceph-mon[25643]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:56:28.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:27 smithi098 ceph-mon[24773]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:56:28.225 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:27 smithi098 ceph-mon[29473]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T03:56:28.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:56:28.642+0000 7f5f8126c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:28.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:56:28.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:28.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:28.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:28.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:28.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:28.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:28.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return future.result(timeout) 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.__get_result() 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: raise self._exception 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:56:28.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:28.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:28.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:28.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:28.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:28.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:28 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:29 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:30.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:30.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:56:30.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:29 smithi153 ceph-mon[25643]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:56:30.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:29 smithi153 ceph-mon[25643]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:56:30.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:29 smithi153 ceph-mon[25643]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:30.324 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.325 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:56:30.326 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:30.327 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:30.328 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:30.329 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:30.330 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:30.331 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:30.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[24773]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[24773]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[24773]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:30.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:30.334 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:30.334 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:30.334 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[29473]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-12T03:56:30.334 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[29473]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T03:56:30.334 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:29 smithi098 ceph-mon[29473]: Reconfiguring daemon prometheus.a on smithi153 2024-03-12T03:56:30.795 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[45390]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:56:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-12T03:56:32.072 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:31 smithi153 ceph-mon[25643]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:56:32.072 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:31 smithi153 systemd[1]: Stopping Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:56:32.072 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:31 smithi153 bash[60509]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:56:32.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:31 smithi098 ceph-mon[24773]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:56:32.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:31 smithi098 ceph-mon[29473]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:56:32.364 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.070Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:56:32.365 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.071Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:56:32.365 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.071Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-12T03:56:32.365 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[57719]: ts=2024-03-12T03:56:32.071Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-12T03:56:32.365 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 podman[60522]: 2024-03-12 03:56:32.080599345 +0000 UTC m=+0.107339754 container died 4f896b5eb96e968702211cbf99e4312dee389b0b91b0d9efa92964f2760dedd5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:32.680 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 podman[60522]: 2024-03-12 03:56:32.46153305 +0000 UTC m=+0.488273459 container cleanup 4f896b5eb96e968702211cbf99e4312dee389b0b91b0d9efa92964f2760dedd5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:32.681 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 bash[60522]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:56:32.681 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 bash[60552]: Error: no container with name or ID "ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus.a" found: no such container 2024-03-12T03:56:32.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 podman[60536]: 2024-03-12 03:56:32.679420328 +0000 UTC m=+0.599986947 container remove 4f896b5eb96e968702211cbf99e4312dee389b0b91b0d9efa92964f2760dedd5 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:32.942 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:56:32.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 systemd[1]: Stopped Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:56:32.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.387s CPU time. 2024-03-12T03:56:32.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 systemd[1]: Starting Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:56:32.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 podman[60616]: 2024-03-12 03:56:32.81525143 +0000 UTC m=+0.019070971 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-12T03:56:32.943 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:32 smithi153 podman[60616]: 2024-03-12 03:56:32.913390022 +0000 UTC m=+0.117209547 container create 250294c7b1c57c6d4dc559c930256be552f0bee0de049f2a888dc5e9b4770464 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:33.360 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 podman[60616]: 2024-03-12 03:56:33.133055124 +0000 UTC m=+0.336874657 container init 250294c7b1c57c6d4dc559c930256be552f0bee0de049f2a888dc5e9b4770464 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:33.360 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 podman[60616]: 2024-03-12 03:56:33.135640984 +0000 UTC m=+0.339460518 container start 250294c7b1c57c6d4dc559c930256be552f0bee0de049f2a888dc5e9b4770464 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.158Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.158Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.158Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.158Z caller=main.go:518 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi153 (none))" 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.158Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.158Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.160Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.160Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.161Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.163Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.163Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.71µs 2024-03-12T03:56:33.361 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.163Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:56:33.362 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 bash[60616]: 250294c7b1c57c6d4dc559c930256be552f0bee0de049f2a888dc5e9b4770464 2024-03-12T03:56:33.362 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 systemd[1]: Started Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:56:33.362 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.358Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-03-12T03:56:33.900 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.613Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-03-12T03:56:33.900 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.844Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-03-12T03:56:33.900 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.899Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-03-12T03:56:34.187 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-mon[25643]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:34.187 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:34.187 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:34.187 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:56:34.187 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:34.187 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.903Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.903Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.903Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=61.595µs wal_replay_duration=740.135732ms total_replay_duration=740.211366ms 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.908Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.908Z caller=main.go:947 level=info msg="TSDB started" 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.908Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.928Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.479353ms db_storage=645ns remote_storage=1.642µs web_handler=366ns query_engine=541ns scrape=447.109µs scrape_sd=101.663µs notify=10.242µs notify_sd=9.369µs rules=19.58776ms 2024-03-12T03:56:34.188 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:56:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:56:33.928Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-12T03:56:34.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[24773]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[29473]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:33 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:34 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:56:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:34 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:56:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:34 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:56:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:34 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:56:35.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:34 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:35.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T03:56:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:34 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:36.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:35 smithi153 ceph-mon[25643]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:36.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:35 smithi153 ceph-mon[25643]: Upgrade: Updating mgr.x 2024-03-12T03:56:36.323 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:35 smithi153 ceph-mon[25643]: Deploying daemon mgr.x on smithi153 2024-03-12T03:56:36.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:35 smithi098 ceph-mon[29473]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:36.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:35 smithi098 ceph-mon[29473]: Upgrade: Updating mgr.x 2024-03-12T03:56:36.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:35 smithi098 ceph-mon[29473]: Deploying daemon mgr.x on smithi153 2024-03-12T03:56:36.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:35 smithi098 ceph-mon[24773]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:36.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:35 smithi098 ceph-mon[24773]: Upgrade: Updating mgr.x 2024-03-12T03:56:36.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:35 smithi098 ceph-mon[24773]: Deploying daemon mgr.x on smithi153 2024-03-12T03:56:36.883 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:36 smithi153 systemd[1]: Stopping Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:56:37.178 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:36 smithi153 podman[60970]: 2024-03-12 03:56:36.881933797 +0000 UTC m=+0.157209535 container died 2e7f2996e113d35dca20c038327a90364ed2a1d144060493cb1c05c54a0b6213 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-12T03:56:37.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 podman[60970]: 2024-03-12 03:56:37.259686646 +0000 UTC m=+0.534962383 container cleanup 2e7f2996e113d35dca20c038327a90364ed2a1d144060493cb1c05c54a0b6213 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:56:37.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 bash[60970]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x 2024-03-12T03:56:37.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 podman[60984]: 2024-03-12 03:56:37.494190378 +0000 UTC m=+0.612600794 container remove 2e7f2996e113d35dca20c038327a90364ed2a1d144060493cb1c05c54a0b6213 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-12T03:56:37.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:56:37.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-12T03:56:37.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 systemd[1]: Stopped Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:56:37.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service: Consumed 57.555s CPU time. 2024-03-12T03:56:37.817 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:37 smithi153 systemd[1]: Starting Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:56:38.162 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:37 smithi153 ceph-mon[25643]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:38.162 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 podman[61119]: 2024-03-12 03:56:37.936084459 +0000 UTC m=+0.018806815 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:56:38.162 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 podman[61119]: 2024-03-12 03:56:38.061779082 +0000 UTC m=+0.144501431 container create bc9c9a5eb6c22d1f07a711acc18f4e6778961745152aa11c068f2c7ec60e619e (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T03:56:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:37 smithi098 ceph-mon[24773]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:37 smithi098 ceph-mon[29473]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T03:56:38.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 podman[61119]: 2024-03-12 03:56:38.272489687 +0000 UTC m=+0.355212036 container init bc9c9a5eb6c22d1f07a711acc18f4e6778961745152aa11c068f2c7ec60e619e (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T03:56:38.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 podman[61119]: 2024-03-12 03:56:38.274983851 +0000 UTC m=+0.357706209 container start bc9c9a5eb6c22d1f07a711acc18f4e6778961745152aa11c068f2c7ec60e619e (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-12T03:56:38.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 ceph-mgr[61139]: -- 172.21.15.153:0/44518334 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e85bec5a0 con 0x557e85bc0800 2024-03-12T03:56:38.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 bash[61119]: bc9c9a5eb6c22d1f07a711acc18f4e6778961745152aa11c068f2c7ec60e619e 2024-03-12T03:56:38.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 systemd[1]: Started Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:56:39.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:38.850+0000 7f892e322200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:56:39.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:38.955+0000 7f892e322200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:56:39.512 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:39 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:39.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:39 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:39.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:39 smithi098 ceph-mon[24773]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:56:39.513 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:39 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:39.513 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:39 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:39.513 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:39 smithi098 ceph-mon[29473]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:56:39.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:39 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:39.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:39 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:39.682 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:39 smithi153 ceph-mon[25643]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:56:39.938 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:39.680+0000 7f892e322200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:56:39.939 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:39.793+0000 7f892e322200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:56:40.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:39.937+0000 7f892e322200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:56:40.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:40.036+0000 7f892e322200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:56:40.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:40.125+0000 7f892e322200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:56:40.443 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:40.403+0000 7f892e322200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:56:40.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:40 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3123398210' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:56:40.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:40 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3123398210' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:56:40.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:56:40] "GET /metrics HTTP/1.1" 200 37918 "" "Prometheus/2.33.4" 2024-03-12T03:56:40.728 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3123398210' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:56:40.728 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:40.527+0000 7f892e322200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:56:40.728 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:40.629+0000 7f892e322200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:56:40.728 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:40.726+0000 7f892e322200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:56:41.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:40.912+0000 7f892e322200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:56:41.570 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:41 smithi153 ceph-mon[25643]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:41.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:41 smithi098 ceph-mon[24773]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:41.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:41 smithi098 ceph-mon[29473]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:42.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:41.918+0000 7f892e322200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:56:42.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:42.065+0000 7f892e322200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:56:42.488 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:42.410+0000 7f892e322200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:56:42.926 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:42.503+0000 7f892e322200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:56:42.927 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:42.713+0000 7f892e322200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:56:43.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:42.924+0000 7f892e322200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:56:43.505 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:43.228+0000 7f892e322200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:56:43.506 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:43.502+0000 7f892e322200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:56:44.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:43 smithi098 ceph-mon[29473]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:44.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:43 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:44.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:43 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:43 smithi098 ceph-mon[24773]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:43 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:44.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:43 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:44.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:43.760+0000 7f892e322200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:56:44.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:43 smithi153 ceph-mon[25643]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:44.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:43 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:43 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:44.865 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:44.630+0000 7f892e322200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:56:45.136 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:44 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:45.136 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:44 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:45.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:44 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:45.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:44 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:45.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:44 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:45.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:44 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:45.419 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:45.219+0000 7f892e322200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:56:45.419 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:45.324+0000 7f892e322200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:56:45.670 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:45.559+0000 7f892e322200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:56:45.670 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: [12/Mar/2024:03:56:45] ENGINE Bus STARTING 2024-03-12T03:56:45.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: CherryPy Checker: 2024-03-12T03:56:45.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: The Application mounted at '' has an empty config. 2024-03-12T03:56:45.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:56:45.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: [12/Mar/2024:03:56:45] ENGINE Serving on http://:::9283 2024-03-12T03:56:45.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: [12/Mar/2024:03:56:45] ENGINE Bus STARTED 2024-03-12T03:56:46.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[24773]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[24773]: Standby manager daemon x restarted 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[24773]: Standby manager daemon x started 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[24773]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[29473]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[29473]: Standby manager daemon x restarted 2024-03-12T03:56:46.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[29473]: Standby manager daemon x started 2024-03-12T03:56:46.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:56:46.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:56:46.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:56:46.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:45 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:56:46.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-mon[25643]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:46.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-mon[25643]: Standby manager daemon x restarted 2024-03-12T03:56:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-mon[25643]: Standby manager daemon x started 2024-03-12T03:56:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:56:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:56:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:56:46.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:45 smithi153 ceph-mon[25643]: from='mgr.? 172.21.15.153:0/1132113861' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:56:47.119 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:46 smithi153 ceph-mon[25643]: mgrmap e41: y(active, since 28s), standbys: x 2024-03-12T03:56:47.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:46 smithi098 ceph-mon[29473]: mgrmap e41: y(active, since 28s), standbys: x 2024-03-12T03:56:47.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:46 smithi098 ceph-mon[24773]: mgrmap e41: y(active, since 28s), standbys: x 2024-03-12T03:56:48.184 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:48.185 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:48.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:48.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:48.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:48.186 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:47 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:48.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:56:48.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:48.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:48.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:56:48.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:47 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:56:49.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:48 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:56:49.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:48 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:56:49.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:48 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:48 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:56:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:48 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:56:49.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:48 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:48 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:56:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:48 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:56:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:48 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:56:50.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:49 smithi098 ceph-mon[24773]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:50.220 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:49 smithi098 ceph-mon[29473]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:50.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:49 smithi153 ceph-mon[25643]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:56:50] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:56:50.437+0000 7f5f8126c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:50.575 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:50.576 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:50.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:50.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:50.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:50.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:50.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:50.577 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return future.result(timeout) 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.__get_result() 2024-03-12T03:56:50.578 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:50.579 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: raise self._exception 2024-03-12T03:56:50.580 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:50.580 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:50.580 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:50.580 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:50.580 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:50.581 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:50.582 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:51.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:51.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.248 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.249 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.250 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:50 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.317 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.318 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.319 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.320 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:51.321 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:50 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:51.466 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (13m) 28s ago 20m 22.7M - 0.23.0 ba2b418f427c ae9380406106 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (12m) 9s ago 20m 58.3M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (3m) 28s ago 20m 51.2M - 3.5 e1d6a67b021e 72e16f5fa11f 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (14s) 9s ago 25m 67.4M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (9m) 28s ago 27m 475M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (27m) 28s ago 27m 69.7M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (25m) 9s ago 25m 54.5M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (25m) 28s ago 25m 56.9M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:56:52.119 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (13m) 28s ago 21m 19.6M - 1.3.1 1dbe0e931976 ff35fa2e462e 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (12m) 9s ago 21m 17.6M - 1.3.1 1dbe0e931976 78fb1d60e454 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (25m) 28s ago 25m 55.7M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (24m) 28s ago 24m 57.4M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (24m) 28s ago 24m 50.3M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (24m) 28s ago 24m 52.0M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (23m) 9s ago 23m 57.7M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (23m) 9s ago 23m 55.0M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (22m) 9s ago 22m 51.6M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (22m) 9s ago 22m 54.4M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (19s) 9s ago 21m 50.0M - 2.33.4 514e6a882f6e 250294c7b1c5 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (20m) 28s ago 20m 76.1M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:56:52.120 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (20m) 9s ago 20m 78.8M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:56:52.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:52.325 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:52.326 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:52.327 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.328 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.329 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:56:52.330 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:52.331 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:52.332 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all mgr 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all crash 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all mds 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all nfs 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all nvmeof 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[29473]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:56:52.333 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all mgr 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all crash 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all mds 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all nfs 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all nvmeof 2024-03-12T03:56:52.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:51 smithi098 ceph-mon[24773]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:56:52.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:52.442 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:52.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:56:52.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:56:52.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:56:52.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all mgr 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all crash 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all mds 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T03:56:52.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all nfs 2024-03-12T03:56:52.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all nvmeof 2024-03-12T03:56:52.448 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:51 smithi153 ceph-mon[25643]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:56:52.521 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:56:52.522 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:56:52.868 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:56:52.868 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T03:56:52.868 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [ 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr" 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "2/2 daemons upgraded", 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:56:52.869 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:56:53.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:52 smithi098 ceph-mon[24773]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:53.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:52 smithi098 ceph-mon[24773]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:53.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:52 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2961313238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:52 smithi098 ceph-mon[29473]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:52 smithi098 ceph-mon[29473]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:52 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2961313238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:53.075 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:52 smithi098 systemd[1]: Stopping Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:56:53.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:52 smithi153 ceph-mon[25643]: from='client.25447 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:53.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:52 smithi153 ceph-mon[25643]: from='client.25453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:53.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:52 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2961313238' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:56:53.525 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:53 smithi098 podman[76779]: 2024-03-12 03:56:53.267514336 +0000 UTC m=+0.151881962 container died ff35fa2e462e9753a01df77fc95dd029cc4af22dfdcda708304730f7e9bd4c33 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:53.823 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:53 smithi098 podman[76779]: 2024-03-12 03:56:53.614853021 +0000 UTC m=+0.499220638 container cleanup ff35fa2e462e9753a01df77fc95dd029cc4af22dfdcda708304730f7e9bd4c33 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:53.824 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:53 smithi098 bash[76779]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a 2024-03-12T03:56:54.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[24773]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:54.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[24773]: Upgrade: Updating node-exporter.a (1/2) 2024-03-12T03:56:54.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[24773]: Deploying daemon node-exporter.a on smithi098 2024-03-12T03:56:54.229 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[24773]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:54.230 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[24773]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:54.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[29473]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:54.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[29473]: Upgrade: Updating node-exporter.a (1/2) 2024-03-12T03:56:54.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[29473]: Deploying daemon node-exporter.a on smithi098 2024-03-12T03:56:54.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[29473]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:54.230 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:53 smithi098 ceph-mon[29473]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:54.230 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:53 smithi098 podman[76791]: 2024-03-12 03:56:53.907841716 +0000 UTC m=+0.640621964 container remove ff35fa2e462e9753a01df77fc95dd029cc4af22dfdcda708304730f7e9bd4c33 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:54.231 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:53 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:56:54.231 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-12T03:56:54.231 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 systemd[1]: Stopped Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:56:54.231 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 5.056s CPU time. 2024-03-12T03:56:54.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:53 smithi153 ceph-mon[25643]: from='client.15558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:54.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:53 smithi153 ceph-mon[25643]: Upgrade: Updating node-exporter.a (1/2) 2024-03-12T03:56:54.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:53 smithi153 ceph-mon[25643]: Deploying daemon node-exporter.a on smithi098 2024-03-12T03:56:54.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:53 smithi153 ceph-mon[25643]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:54.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:53 smithi153 ceph-mon[25643]: from='client.25471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:56:54.573 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 systemd[1]: Starting Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:56:54.574 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 bash[76927]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T03:56:55.073 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 bash[76927]: Getting image source signatures 2024-03-12T03:56:55.073 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 bash[76927]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T03:56:55.074 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 bash[76927]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T03:56:55.074 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:54 smithi098 bash[76927]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T03:56:56.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:55 smithi098 ceph-mon[24773]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:56.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:55 smithi098 ceph-mon[29473]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:56.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:55 smithi153 ceph-mon[25643]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:56.823 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:56 smithi098 bash[76927]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:56:56.823 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:56 smithi098 bash[76927]: Writing manifest to image destination 2024-03-12T03:56:57.180 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:56 smithi098 podman[76927]: 2024-03-12 03:56:56.802643321 +0000 UTC m=+2.476885891 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:56:57.180 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:56 smithi098 podman[76927]: 2024-03-12 03:56:56.909356077 +0000 UTC m=+2.583598629 container create ba4b1fd77bbb231b13261a57efb1ab9cddcc6ded013bc47c9ed7818b95d840de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:57.574 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 podman[76927]: 2024-03-12 03:56:57.178922311 +0000 UTC m=+2.853164863 container init ba4b1fd77bbb231b13261a57efb1ab9cddcc6ded013bc47c9ed7818b95d840de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 podman[76927]: 2024-03-12 03:56:57.182041228 +0000 UTC m=+2.856283773 container start ba4b1fd77bbb231b13261a57efb1ab9cddcc6ded013bc47c9ed7818b95d840de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.187Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.187Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.188Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.188Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.188Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.188Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:56:57.575 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:56:57.576 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:56:57.576 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:56:57.576 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:56:57.576 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:56:57.576 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:56:57.577 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:56:57.578 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:56:57.579 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:56:57.580 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:56:57.580 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:56:57.580 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.189Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:56:57.580 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.190Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:56:57.580 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a[76979]: ts=2024-03-12T03:56:57.190Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:56:57.580 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 bash[76927]: ba4b1fd77bbb231b13261a57efb1ab9cddcc6ded013bc47c9ed7818b95d840de 2024-03-12T03:56:57.580 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 03:56:57 smithi098 systemd[1]: Started Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:56:58.230 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:57 smithi153 ceph-mon[25643]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:58.230 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:57 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:58.230 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:57 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-mon[24773]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-mon[29473]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1 op/s 2024-03-12T03:56:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:57 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:56:59.692 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:56:59 smithi153 systemd[1]: Stopping Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:00.070 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:59 smithi153 ceph-mon[25643]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:00.071 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:59 smithi153 ceph-mon[25643]: Upgrade: Updating node-exporter.b (2/2) 2024-03-12T03:57:00.071 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:56:59 smithi153 ceph-mon[25643]: Deploying daemon node-exporter.b on smithi153 2024-03-12T03:57:00.071 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:56:59 smithi153 podman[62515]: 2024-03-12 03:56:59.780251035 +0000 UTC m=+0.110540157 container died 78fb1d60e45411315445059920e95429426d7be9601168415e1bc80470c6953c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:57:00.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:57:00] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-03-12T03:57:00.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:59 smithi098 ceph-mon[24773]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:00.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:59 smithi098 ceph-mon[24773]: Upgrade: Updating node-exporter.b (2/2) 2024-03-12T03:57:00.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:56:59 smithi098 ceph-mon[24773]: Deploying daemon node-exporter.b on smithi153 2024-03-12T03:57:00.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:59 smithi098 ceph-mon[29473]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:00.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:59 smithi098 ceph-mon[29473]: Upgrade: Updating node-exporter.b (2/2) 2024-03-12T03:57:00.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:56:59 smithi098 ceph-mon[29473]: Deploying daemon node-exporter.b on smithi153 2024-03-12T03:57:00.366 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 podman[62515]: 2024-03-12 03:57:00.069306496 +0000 UTC m=+0.399595610 container cleanup 78fb1d60e45411315445059920e95429426d7be9601168415e1bc80470c6953c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:57:00.366 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 bash[62515]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b 2024-03-12T03:57:00.366 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 podman[62530]: 2024-03-12 03:57:00.278932169 +0000 UTC m=+0.502820077 container remove 78fb1d60e45411315445059920e95429426d7be9601168415e1bc80470c6953c (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:57:00.366 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-12T03:57:00.692 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-12T03:57:00.692 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 systemd[1]: Stopped Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:57:00.692 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 5.156s CPU time. 2024-03-12T03:57:00.692 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 systemd[1]: Starting Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:01.157 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:00 smithi153 bash[62659]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-12T03:57:01.442 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:01 smithi153 bash[62659]: Getting image source signatures 2024-03-12T03:57:01.442 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:01 smithi153 bash[62659]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-12T03:57:01.442 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:01 smithi153 bash[62659]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-12T03:57:01.442 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:01 smithi153 bash[62659]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-12T03:57:02.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:02 smithi098 ceph-mon[24773]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:02.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:02 smithi098 ceph-mon[29473]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:02.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:02 smithi153 ceph-mon[25643]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:02.942 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:02 smithi153 bash[62659]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-12T03:57:02.942 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:02 smithi153 bash[62659]: Writing manifest to image destination 2024-03-12T03:57:03.278 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3875034401' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:03.278 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:02 smithi153 podman[62659]: 2024-03-12 03:57:02.99237199 +0000 UTC m=+2.318412941 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-12T03:57:03.278 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 podman[62659]: 2024-03-12 03:57:03.083194749 +0000 UTC m=+2.409235729 container create a0fe78879ec55c03c79c8b0c78a0f7a9f10e99eec464d01e2e83d876c2cbe19c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:57:03.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:03 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3875034401' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:03.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:03 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3875034401' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:03.540 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 podman[62659]: 2024-03-12 03:57:03.276783925 +0000 UTC m=+2.602824867 container init a0fe78879ec55c03c79c8b0c78a0f7a9f10e99eec464d01e2e83d876c2cbe19c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 podman[62659]: 2024-03-12 03:57:03.279625549 +0000 UTC m=+2.605666491 container start a0fe78879ec55c03c79c8b0c78a0f7a9f10e99eec464d01e2e83d876c2cbe19c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.283Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.283Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=arp 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-12T03:57:03.541 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-12T03:57:03.542 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-12T03:57:03.542 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-12T03:57:03.542 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-12T03:57:03.542 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.284Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-12T03:57:03.542 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=edac 2024-03-12T03:57:03.542 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-12T03:57:03.542 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-12T03:57:03.543 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=os 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=stat 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=time 2024-03-12T03:57:03.544 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=uname 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b[62708]: ts=2024-03-12T03:57:03.285Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 bash[62659]: a0fe78879ec55c03c79c8b0c78a0f7a9f10e99eec464d01e2e83d876c2cbe19c 2024-03-12T03:57:03.545 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 03:57:03 smithi153 systemd[1]: Started Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:57:04.303 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:04 smithi153 ceph-mon[25643]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:04.304 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:04 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:04.304 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:04 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:04.304 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:04 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:04.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[24773]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:04.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:04.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:04.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:04.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[29473]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:04.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:04.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:04.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:04 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:06.300 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:06 smithi153 ceph-mon[25643]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:06.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:06 smithi098 ceph-mon[29473]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:06.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:06 smithi098 ceph-mon[24773]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:08.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[24773]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[29473]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:08 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:08 smithi153 ceph-mon[25643]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:08.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:08 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:08 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:08 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:08.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:08 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:09 smithi153 ceph-mon[25643]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.731 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.731 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.731 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[29473]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[24773]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:09.732 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:10.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:57:10] "GET /metrics HTTP/1.1" 200 37912 "" "Prometheus/2.33.4" 2024-03-12T03:57:11.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:11 smithi098 ceph-mon[24773]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:11.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:11 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:11.909 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:11 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:11.910 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:11 smithi098 ceph-mon[29473]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:11.910 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:11 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:11.910 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:11 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:11 smithi153 ceph-mon[25643]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:11 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:11 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='client.25486 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:13.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:13.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:13.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.024 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='client.25486 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:13.025 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='client.25486 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:57:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:14.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:13 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:14.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:13 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:14.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:13 smithi098 ceph-mon[24773]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:14.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:13 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:14.050 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:13 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:14.050 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:13 smithi098 ceph-mon[29473]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:14.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:13 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:14.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:13 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:14.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:13 smithi153 ceph-mon[25643]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:15.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:15.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:57:15.632+0000 7f5f8126c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:15.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:57:15.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:15.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:15.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:15.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return future.result(timeout) 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:15.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.__get_result() 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: raise self._exception 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:15.829 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:15.830 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:15 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:16.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-12T03:57:16.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:16.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:16.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:15 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:17.018 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:16 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:17.018 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:17.018 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:17.018 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:17.018 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:17.019 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:57:17.020 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:17.021 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:17.022 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:17.023 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:17.023 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.023 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:16 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:17.074 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:17.075 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:17.076 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:17.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:17.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:17.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:17.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:17.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:17.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:17.077 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:17.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:17.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:17.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:16 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:17.080 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:17.081 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:17.082 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:17.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:17.084 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:17.084 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:17.084 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:17.084 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:17.084 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:17.084 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:18.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:17 smithi098 ceph-mon[24773]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:18.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:17 smithi098 ceph-mon[24773]: Upgrade: Updating prometheus.a 2024-03-12T03:57:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:17 smithi098 ceph-mon[29473]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:18.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:17 smithi098 ceph-mon[29473]: Upgrade: Updating prometheus.a 2024-03-12T03:57:18.110 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:17 smithi153 ceph-mon[25643]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:57:18.110 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:17 smithi153 ceph-mon[25643]: Upgrade: Updating prometheus.a 2024-03-12T03:57:18.934 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:18 smithi153 ceph-mon[25643]: Deploying daemon prometheus.a on smithi153 2024-03-12T03:57:19.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:18 smithi098 ceph-mon[24773]: Deploying daemon prometheus.a on smithi153 2024-03-12T03:57:19.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:18 smithi098 ceph-mon[29473]: Deploying daemon prometheus.a on smithi153 2024-03-12T03:57:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:19 smithi153 ceph-mon[25643]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:57:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:19 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:20.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:19 smithi098 ceph-mon[24773]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:57:20.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:19 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:19 smithi098 ceph-mon[29473]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:57:20.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:19 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:57:20] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.33.4" 2024-03-12T03:57:20.931 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 systemd[1]: Stopping Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.929Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.929Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.929Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.929Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.929Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.929Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.930Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.930Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T03:57:21.193 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.930Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T03:57:21.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.930Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T03:57:21.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.930Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-12T03:57:21.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[60631]: ts=2024-03-12T03:57:20.930Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-12T03:57:21.194 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:20 smithi153 podman[64072]: 2024-03-12 03:57:20.940052538 +0000 UTC m=+0.126669771 container died 250294c7b1c57c6d4dc559c930256be552f0bee0de049f2a888dc5e9b4770464 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:21.539 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:21 smithi153 podman[64072]: 2024-03-12 03:57:21.303976098 +0000 UTC m=+0.490593337 container cleanup 250294c7b1c57c6d4dc559c930256be552f0bee0de049f2a888dc5e9b4770464 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:21.540 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:21 smithi153 bash[64072]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T03:57:21.856 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:21 smithi153 podman[64086]: 2024-03-12 03:57:21.53855212 +0000 UTC m=+0.601008136 container remove 250294c7b1c57c6d4dc559c930256be552f0bee0de049f2a888dc5e9b4770464 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:21.856 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:21 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-12T03:57:21.856 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:21 smithi153 systemd[1]: Stopped Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:57:21.856 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:21 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.276s CPU time. 2024-03-12T03:57:21.856 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:21 smithi153 ceph-mon[25643]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:57:22.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:21 smithi098 ceph-mon[24773]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:57:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:21 smithi098 ceph-mon[29473]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-12T03:57:22.185 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:21 smithi153 systemd[1]: Starting Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:22.185 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 podman[64217]: 2024-03-12 03:57:21.973770438 +0000 UTC m=+0.018908798 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-12T03:57:22.185 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 podman[64217]: 2024-03-12 03:57:22.081619651 +0000 UTC m=+0.126758049 container create 7597e44b0bb5de73778cb090d5517f781735e2e0f132087f9f9bffc8a47eb8e8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:22.692 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 podman[64217]: 2024-03-12 03:57:22.292501115 +0000 UTC m=+0.337639482 container init 7597e44b0bb5de73778cb090d5517f781735e2e0f132087f9f9bffc8a47eb8e8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 podman[64217]: 2024-03-12 03:57:22.294974416 +0000 UTC m=+0.340112777 container start 7597e44b0bb5de73778cb090d5517f781735e2e0f132087f9f9bffc8a47eb8e8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.318Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.318Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.318Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi153 (none))" 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.318Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.318Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.320Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.320Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.322Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-12T03:57:22.693 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.322Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.324Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.324Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.67µs 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.324Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.331Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.344Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.354Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.357Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.360Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.361Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.361Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.361Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.426µs wal_replay_duration=37.192469ms wbl_replay_duration=99ns total_replay_duration=37.237313ms 2024-03-12T03:57:22.694 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.364Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-12T03:57:22.695 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.364Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-12T03:57:22.695 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.364Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-12T03:57:22.695 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 bash[64217]: 7597e44b0bb5de73778cb090d5517f781735e2e0f132087f9f9bffc8a47eb8e8 2024-03-12T03:57:22.695 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 systemd[1]: Started Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:57:22.695 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.390Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.278442ms db_storage=763ns remote_storage=1.22µs web_handler=394ns query_engine=677ns scrape=419.806µs scrape_sd=79.552µs notify=16.765µs notify_sd=16.819µs rules=24.363771ms tracing=6.82µs 2024-03-12T03:57:22.695 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.390Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-12T03:57:22.695 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 03:57:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T03:57:22.390Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-12T03:57:23.221 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:57:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:23 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:23 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:23 smithi153 ceph-mon[25643]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:23 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:23 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:23 smithi098 ceph-mon[24773]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:23 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:23 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:23 smithi098 ceph-mon[29473]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:23.866 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:57:23.866 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (13m) 15s ago 21m 22.7M - 0.23.0 ba2b418f427c ae9380406106 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (13m) 16s ago 21m 57.4M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (22s) 15s ago 20m 45.9M - 3.5 e1d6a67b021e c65a7775297b 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (45s) 16s ago 26m 450M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (10m) 15s ago 27m 498M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (27m) 15s ago 27m 73.9M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (26m) 16s ago 26m 56.0M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (26m) 15s ago 26m 57.8M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (27s) 15s ago 22m 5452k - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (20s) 16s ago 22m 3577k - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (25m) 15s ago 25m 56.4M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (25m) 15s ago 25m 58.6M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (24m) 15s ago 24m 50.4M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (24m) 15s ago 24m 52.9M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:57:23.867 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (24m) 16s ago 24m 59.4M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:57:23.868 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (23m) 16s ago 23m 54.9M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:57:23.868 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (23m) 16s ago 23m 52.8M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:57:23.868 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (23m) 16s ago 23m 55.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:57:23.868 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 starting - - - - 2024-03-12T03:57:23.868 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (20m) 15s ago 20m 76.4M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:57:23.868 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (20m) 16s ago 20m 78.9M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:57:24.252 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:57:24.252 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:57:24.253 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:24.254 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:57:24.254 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:57:24.254 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:57:24.254 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:57:24.254 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [ 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr" 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "2/2 daemons upgraded", 2024-03-12T03:57:24.581 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-12T03:57:24.582 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:57:24.582 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:57:25.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:24 smithi153 ceph-mon[25643]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:24 smithi153 ceph-mon[25643]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:24 smithi153 ceph-mon[25643]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.030 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:24 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/3773649821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:25.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[24773]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[24773]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[24773]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3773649821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[29473]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[29473]: from='client.15606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[29473]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:24 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3773649821' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:25 smithi153 ceph-mon[25643]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:25 smithi153 ceph-mon[25643]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:26.067 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:25 smithi098 ceph-mon[29473]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:26.068 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:25 smithi098 ceph-mon[29473]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:26.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:25 smithi098 ceph-mon[24773]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:26.068 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:25 smithi098 ceph-mon[24773]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:28.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[24773]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/218800691' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[29473]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:28.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:27 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/218800691' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:27 smithi153 ceph-mon[25643]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T03:57:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:27 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:27 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:27 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/218800691' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:28 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:28 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:29.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:28 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:29.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:28 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:28 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:29.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:28 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:29.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:29 smithi153 ceph-mon[25643]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:30.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:29 smithi098 ceph-mon[24773]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:30.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:29 smithi098 ceph-mon[29473]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:57:30] "GET /metrics HTTP/1.1" 200 37914 "" "Prometheus/2.43.0" 2024-03-12T03:57:32.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:32.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:32 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:32.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:32.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:32.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:32 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:33 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:33 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:33.450 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:33 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:33.450 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:33 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:33.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:33 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:33.451 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:33 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:34.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-mon[24773]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:34.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-mon[29473]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:34.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:34 smithi153 ceph-mon[25643]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:34.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:34 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:57:34.356+0000 7f5f8126c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:57:34.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:34.825 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return future.result(timeout) 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:34.826 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.__get_result() 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: raise self._exception 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:34.827 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:34.828 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:34 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:35.196 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.197 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.198 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:35 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:57:35.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:35.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:35.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:35 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:36.365 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:36 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:36.366 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:36.367 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:36.368 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:36.369 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:36 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:36 smithi098 ceph-mon[24773]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:36 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:36.370 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:36.371 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:36.372 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.373 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.374 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:36 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:57:36.375 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:36 smithi098 ceph-mon[29473]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:36.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:36 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:36.443 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:36.444 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:36.445 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:36.446 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:36.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:36.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:36.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:36.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:36.447 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:36.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:36 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-12T03:57:36.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:36 smithi153 ceph-mon[25643]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T03:57:37.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:37 smithi153 ceph-mon[25643]: Upgrade: Updating alertmanager.a 2024-03-12T03:57:37.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:37 smithi153 ceph-mon[25643]: Deploying daemon alertmanager.a on smithi098 2024-03-12T03:57:37.541 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:37 smithi098 ceph-mon[24773]: Upgrade: Updating alertmanager.a 2024-03-12T03:57:37.542 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:37 smithi098 ceph-mon[24773]: Deploying daemon alertmanager.a on smithi098 2024-03-12T03:57:37.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:37 smithi098 ceph-mon[29473]: Upgrade: Updating alertmanager.a 2024-03-12T03:57:37.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:37 smithi098 ceph-mon[29473]: Deploying daemon alertmanager.a on smithi098 2024-03-12T03:57:38.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:38 smithi098 ceph-mon[24773]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:38.362 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:38 smithi098 ceph-mon[29473]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:38.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:38 smithi153 ceph-mon[25643]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:39.531 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:39 smithi098 systemd[1]: Stopping Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:39.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:39 smithi098 ceph-mon[24773]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:39 smithi098 ceph-mon[29473]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:39.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:39 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[58998]: level=info ts=2024-03-12T03:57:39.528Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-12T03:57:39.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:39 smithi098 podman[79356]: 2024-03-12 03:57:39.569590878 +0000 UTC m=+0.157337011 container died ae93804061060426dd9dc3bb7a467194cbdfbf6744047ea7e6b81c507955acc7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:40.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:39 smithi153 ceph-mon[25643]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:40.214 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:39 smithi098 podman[79356]: 2024-03-12 03:57:39.954026417 +0000 UTC m=+0.541772552 container cleanup ae93804061060426dd9dc3bb7a467194cbdfbf6744047ea7e6b81c507955acc7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:40.214 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:39 smithi098 bash[79356]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T03:57:40.473 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:57:40] "GET /metrics HTTP/1.1" 200 37915 "" "Prometheus/2.43.0" 2024-03-12T03:57:40.473 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 podman[79370]: 2024-03-12 03:57:40.213291549 +0000 UTC m=+0.633340063 container remove ae93804061060426dd9dc3bb7a467194cbdfbf6744047ea7e6b81c507955acc7 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:40.473 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 podman[79370]: 2024-03-12 03:57:40.314046877 +0000 UTC m=+0.734095401 volume remove 43f5f9a50572ba1c177c6dcb6001e8987e29de5da7d0f758783ae27bd531ed12 2024-03-12T03:57:40.473 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T03:57:40.474 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 systemd[1]: Stopped Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:57:40.474 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.470s CPU time. 2024-03-12T03:57:40.823 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 systemd[1]: Starting Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:41.169 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 podman[79502]: 2024-03-12 03:57:40.766292624 +0000 UTC m=+0.018325226 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-12T03:57:41.169 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 podman[79502]: 2024-03-12 03:57:40.890445981 +0000 UTC m=+0.142478583 volume create 2758b10a68e464d14a66a36f64df6422801ad2f52ee17db5faae5813dc450031 2024-03-12T03:57:41.169 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:40 smithi098 podman[79502]: 2024-03-12 03:57:40.982538749 +0000 UTC m=+0.234571396 container create 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 podman[79502]: 2024-03-12 03:57:41.277097454 +0000 UTC m=+0.529130070 container init 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 podman[79502]: 2024-03-12 03:57:41.279902017 +0000 UTC m=+0.531934618 container start 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.292Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.292Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.293Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.98 port=9094 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.295Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.318Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.318Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.321Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:41.321Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-12T03:57:41.574 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 bash[79502]: 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 2024-03-12T03:57:41.575 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:41 smithi098 systemd[1]: Started Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:57:42.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-mon[29473]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:42.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:42.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:42.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-mon[24773]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:42.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:42.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:41 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:42.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:41 smithi153 ceph-mon[25643]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 0 op/s 2024-03-12T03:57:42.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:41 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:42.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:41 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:43.573 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:43.295Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000256745s 2024-03-12T03:57:43.976 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:43 smithi153 ceph-mon[25643]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:44.054 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:43 smithi098 ceph-mon[24773]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:44.055 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:43 smithi098 ceph-mon[29473]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:46.008 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:45 smithi098 ceph-mon[24773]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:46.008 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:45 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:46.008 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:45 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:46.008 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:45 smithi098 ceph-mon[29473]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:46.009 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:45 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:46.009 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:45 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:45 smithi153 ceph-mon[25643]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:45 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:45 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:46 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:46 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:46 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/4000747207' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:47.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:46 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:47.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:46 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:47.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:46 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/4000747207' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:46 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:46 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:47.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:46 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/4000747207' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:57:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:47 smithi098 ceph-mon[24773]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:47 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:48.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:47 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:48.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:47 smithi098 ceph-mon[29473]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:48.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:47 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:48.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:47 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:47 smithi153 ceph-mon[25643]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:57:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:47 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:48.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:47 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:49.998 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:49 smithi098 ceph-mon[24773]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:49.999 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:49 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:49.999 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:49 smithi098 ceph-mon[29473]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:49.999 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:49 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:49 smithi153 ceph-mon[25643]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:57:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:49 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:57:50.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:57:50] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.43.0" 2024-03-12T03:57:51.387 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:57:51.297Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001865699s 2024-03-12T03:57:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:51 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:51.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:57:51.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:51 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:57:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:52 smithi153 ceph-mon[25643]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-12T03:57:52.743 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:52 smithi098 ceph-mon[24773]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-12T03:57:52.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:52 smithi098 ceph-mon[29473]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-12T03:57:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:53 smithi153 ceph-mon[25643]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-mon[24773]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-mon[29473]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:54.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:57:53.909+0000 7f5f8126c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:54.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:57:54.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:54.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:54.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:54.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:54.325 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:54.326 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.326 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.326 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:54.326 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:54.326 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:54.326 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return future.result(timeout) 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.__get_result() 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: raise self._exception 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:54.327 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:54.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:54.329 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:54.329 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:54.329 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:54.329 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:54.329 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.329 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:57:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.837 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:54.839 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:54.840 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:54.841 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:54.842 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:54.943 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:55.074 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.075 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:55.076 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:55.077 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:55.078 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:55.079 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:57:55.080 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:57:55.081 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:57:55.082 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:57:55.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:57:55.084 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (14s) 9s ago 22m 15.2M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (13m) 10s ago 21m 57.8M - 8.3.5 dad864ee21e9 9057d202c6a7 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 unknown 9s ago 21m - - 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (77s) 10s ago 26m 450M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (11m) 9s ago 28m 501M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (28m) 9s ago 28m 72.2M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (26m) 10s ago 26m 58.0M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (26m) 9s ago 26m 59.2M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (58s) 9s ago 22m 8396k - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T03:57:55.587 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (52s) 10s ago 22m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (26m) 9s ago 26m 56.4M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (25m) 9s ago 25m 58.6M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (25m) 9s ago 25m 50.5M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (25m) 9s ago 25m 52.9M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (24m) 10s ago 24m 59.4M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (24m) 10s ago 24m 54.9M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (23m) 10s ago 23m 52.8M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (23m) 10s ago 23m 55.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (33s) 10s ago 22m 53.2M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (21m) 9s ago 21m 76.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:57:55.588 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (21m) 10s ago 21m 79.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:57:55.980 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:57:55.981 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:57:55.982 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:55 smithi098 ceph-mon[24773]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:55.982 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:55 smithi098 ceph-mon[24773]: from='client.25579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:55.982 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:55 smithi098 ceph-mon[29473]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:55.982 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:55 smithi098 ceph-mon[29473]: from='client.25579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:56.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:55 smithi153 ceph-mon[25643]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 350 B/s rd, 0 op/s 2024-03-12T03:57:56.067 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:55 smithi153 ceph-mon[25643]: from='client.25579 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [ 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr" 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "2/2 daemons upgraded", 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading grafana daemons", 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:57:56.304 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:57:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:56 smithi153 ceph-mon[25643]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:56 smithi153 ceph-mon[25643]: Upgrade: Updating grafana.a 2024-03-12T03:57:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:56 smithi153 ceph-mon[25643]: Deploying daemon grafana.a on smithi153 2024-03-12T03:57:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:56 smithi153 ceph-mon[25643]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:57.006 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:56 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2774035990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:57.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[24773]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:57.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[24773]: Upgrade: Updating grafana.a 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[24773]: Deploying daemon grafana.a on smithi153 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[24773]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2774035990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[29473]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[29473]: Upgrade: Updating grafana.a 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[29473]: Deploying daemon grafana.a on smithi153 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[29473]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:57.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:56 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2774035990' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:57:58.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:57 smithi153 ceph-mon[25643]: from='client.25603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:58.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:57 smithi153 ceph-mon[25643]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:57:58.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:57 smithi098 ceph-mon[24773]: from='client.25603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:58.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:57 smithi098 ceph-mon[24773]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:57:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:57 smithi098 ceph-mon[29473]: from='client.25603 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:57:58.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:57 smithi098 ceph-mon[29473]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:57:58.536 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:58 smithi153 systemd[1]: Stopping Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:58.536 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[49713]: t=2024-03-12T03:57:58+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-12T03:57:58.850 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:58 smithi153 podman[66188]: 2024-03-12 03:57:58.534508568 +0000 UTC m=+0.110889275 container died 9057d202c6a75776c0a39a39c0fe4005fccfbed59393ce506c5875774bf52907 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, io.openshift.expose-services=, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, release=236.1648460182, architecture=x86_64, io.openshift.tags=base rhel8, description=Ceph Grafana Container, distribution-scope=public, version=8.5, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, vcs-type=git) 2024-03-12T03:57:59.192 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:58 smithi153 podman[66188]: 2024-03-12 03:57:58.848812384 +0000 UTC m=+0.425193087 container cleanup 9057d202c6a75776c0a39a39c0fe4005fccfbed59393ce506c5875774bf52907 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, architecture=x86_64, name=ubi8, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , description=Ceph Grafana Container, vcs-type=git, version=8.5) 2024-03-12T03:57:59.192 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:58 smithi153 bash[66188]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a 2024-03-12T03:57:59.192 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:59 smithi153 podman[66202]: 2024-03-12 03:57:59.033380825 +0000 UTC m=+0.496412996 container remove 9057d202c6a75776c0a39a39c0fe4005fccfbed59393ce506c5875774bf52907 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., maintainer=Paul Cuzner , release=236.1648460182, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, name=ubi8, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, io.buildah.version=1.24.2, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public) 2024-03-12T03:57:59.673 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:59 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-12T03:57:59.673 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:59 smithi153 systemd[1]: Stopped Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:57:59.673 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:59 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service: Consumed 5.343s CPU time. 2024-03-12T03:57:59.673 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:59 smithi153 systemd[1]: Starting Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:57:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:57:59 smithi153 ceph-mon[25643]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:57:59.942 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:59 smithi153 podman[66332]: 2024-03-12 03:57:59.57344211 +0000 UTC m=+0.017753294 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-12T03:57:59.942 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:57:59 smithi153 podman[66332]: 2024-03-12 03:57:59.701662877 +0000 UTC m=+0.145974051 container create 499a74c577ce46021f3273f2a6d6114c5583b4e005d2e635bf053f253cd72f90 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.openshift.expose-services=) 2024-03-12T03:58:00.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:57:59 smithi098 ceph-mon[24773]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:58:00.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:57:59 smithi098 ceph-mon[29473]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 podman[66332]: 2024-03-12 03:58:00.029691633 +0000 UTC m=+0.474002812 container init 499a74c577ce46021f3273f2a6d6114c5583b4e005d2e635bf053f253cd72f90 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, description=Ceph Grafana Container, vendor=Red Hat, Inc., vcs-type=git, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, architecture=x86_64, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 podman[66332]: 2024-03-12 03:58:00.032571917 +0000 UTC m=+0.476883091 container start 499a74c577ce46021f3273f2a6d6114c5583b4e005d2e635bf053f253cd72f90 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, description=Ceph Grafana Container, version=8.5, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.expose-services=, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, name=ubi8) 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 bash[66332]: 499a74c577ce46021f3273f2a6d6114c5583b4e005d2e635bf053f253cd72f90 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 systemd[1]: Started Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186118256Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186321362Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186340885Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-12T03:58:00.199 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186348488Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186355081Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186361531Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186368115Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186373959Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186379861Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186385444Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186390834Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186396241Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186401811Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=settings t=2024-03-12T03:58:00.186407469Z level=info msg="App mode production" 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=sqlstore t=2024-03-12T03:58:00.186449156Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=sqlstore t=2024-03-12T03:58:00.186475116Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-12T03:58:00.200 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.197640962Z level=info msg="Starting DB migrations" 2024-03-12T03:58:00.478 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.201361894Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-12T03:58:00.478 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.210190559Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-12T03:58:00.478 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.21123875Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-12T03:58:00.478 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.211579144Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-12T03:58:00.478 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.212732532Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.213749936Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.214691307Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.215537074Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.216025858Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.216930409Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.217727205Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.218679852Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.219565958Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.220527199Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.221471534Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.222408272Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.22522549Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-12T03:58:00.479 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.227797809Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-12T03:58:00.480 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.230504443Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-12T03:58:00.480 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.230830406Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-12T03:58:00.480 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.233573993Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-12T03:58:00.480 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.236388279Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-12T03:58:00.480 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.23687047Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.239559424Z level=info msg="Executing migration" id="create alert_image table" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.242233Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.24515297Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.245474181Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.248146141Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.251000098Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.251095077Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.253880059Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.254405852Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.257011691Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.25960553Z level=info msg="Executing migration" id="create secrets table" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.262430539Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-12T03:58:00.481 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.267259559Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.269851629Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.272623373Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.277696648Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.282519238Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.284915402Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.287438407Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.290041772Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.290356244Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.292772225Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.295442789Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.297980242Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.298352322Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.301107176Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-12T03:58:00.482 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.303627832Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.306273204Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.307342768Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.308214102Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.309030799Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.309787995Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.310249418Z level=info msg="Executing migration" id="add correlation config column" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.311228034Z level=info msg="Executing migration" id="create entity_events table" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.312036996Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.312845236Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.312939169Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.313222109Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.313305054Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.314080295Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-12T03:58:00.483 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.314911119Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.315658008Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.316161429Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.316967342Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.317790584Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.318633532Z level=info msg="Executing migration" id="Drop public config table" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.319085581Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.319856613Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.320339532Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-12T03:58:00.484 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.320811588Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.32159189Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.324663066Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.325755329Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.326786234Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.327584486Z level=info msg="Executing migration" id="add share column" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.328639285Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.329396917Z level=info msg="Executing migration" id="create file table" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.330264912Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.331143315Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-12T03:58:00.485 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.33158207Z level=info msg="Executing migration" id="create file_meta table" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.33242111Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.33332276Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.334170119Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.335359584Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.336133066Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.336982871Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.339927329Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.340284037Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.342880222Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.345664089Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.348397849Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.348881384Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.3513715Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-12T03:58:00.486 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.353948936Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.356519433Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.366433502Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.369162655Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.371833326Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.375359463Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.375993388Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.378737021Z level=info msg="Executing migration" id="create folder table" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.381268374Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.38401609Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.38451205Z level=info msg="Executing migration" id="Update folder title length" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.387272429Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=migrator t=2024-03-12T03:58:00.389747324Z level=info msg="migrations completed" performed=105 skipped=376 duration=188.435066ms 2024-03-12T03:58:00.487 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=sqlstore t=2024-03-12T03:58:00.390062245Z level=info msg="Created default organization" 2024-03-12T03:58:00.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=plugin.loader t=2024-03-12T03:58:00.447521741Z level=info msg="Plugin registered" pluginID=input 2024-03-12T03:58:00.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=plugin.loader t=2024-03-12T03:58:00.476237361Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-12T03:58:00.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=plugin.loader t=2024-03-12T03:58:00.476373123Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-12T03:58:00.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=secrets t=2024-03-12T03:58:00.476475559Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-12T03:58:00.488 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=query_data t=2024-03-12T03:58:00.480721283Z level=info msg="Query Service initialization" 2024-03-12T03:58:00.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:58:00] "GET /metrics HTTP/1.1" 200 37911 "" "Prometheus/2.43.0" 2024-03-12T03:58:00.942 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:00 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=live.push_http t=2024-03-12T03:58:00.483195565Z level=info msg="Live Push Gateway initialization" 2024-03-12T03:58:01.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:01.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:01.204 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-mon[25643]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=infra.usagestats.collector t=2024-03-12T03:58:01.136090277Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=provisioning.datasources t=2024-03-12T03:58:01.138070461Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=provisioning.datasources t=2024-03-12T03:58:01.138087871Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=provisioning.datasources t=2024-03-12T03:58:01.138213861Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=provisioning.alerting t=2024-03-12T03:58:01.160326783Z level=info msg="starting to provision alerting" 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=provisioning.alerting t=2024-03-12T03:58:01.160347752Z level=info msg="finished to provision alerting" 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=grafanaStorageLogger t=2024-03-12T03:58:01.160488245Z level=info msg="storage starting" 2024-03-12T03:58:01.204 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=ngalert.state.manager t=2024-03-12T03:58:01.160602137Z level=info msg="Warming state cache for startup" 2024-03-12T03:58:01.205 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=ngalert.state.manager t=2024-03-12T03:58:01.160812935Z level=info msg="State cache has been initialized" states=0 duration=209.936µs 2024-03-12T03:58:01.205 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=ticker t=2024-03-12T03:58:01.160928631Z level=info msg=starting first_tick=2024-03-12T03:58:10Z 2024-03-12T03:58:01.205 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=ngalert.multiorg.alertmanager t=2024-03-12T03:58:01.160986682Z level=info msg="starting MultiOrg Alertmanager" 2024-03-12T03:58:01.205 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=http.server t=2024-03-12T03:58:01.162979903Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-12T03:58:01.205 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=sqlstore.transactions t=2024-03-12T03:58:01.179093281Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-12T03:58:01.205 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=sqlstore.transactions t=2024-03-12T03:58:01.190226079Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-12T03:58:01.205 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 03:58:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=sqlstore.transactions t=2024-03-12T03:58:01.202516994Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-12T03:58:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:01 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:01 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:01 smithi098 ceph-mon[24773]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:58:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:01 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:01 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:01 smithi098 ceph-mon[29473]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 700 B/s rd, 0 op/s 2024-03-12T03:58:03.487 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:03 smithi153 ceph-mon[25643]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:03.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:03 smithi098 ceph-mon[24773]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:03 smithi098 ceph-mon[29473]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:05.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:04 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:04 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:04 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:04 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:04 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:06.050 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:05 smithi153 ceph-mon[25643]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:06.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:05 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:06.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:05 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:06.051 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:05 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2937194981' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:58:06.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[24773]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:06.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:06.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:06.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2937194981' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:58:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[29473]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:05 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2937194981' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:58:08.060 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:07 smithi153 ceph-mon[25643]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:08.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:07 smithi098 ceph-mon[24773]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:08.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:07 smithi098 ceph-mon[29473]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T03:58:10.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:58:10.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:10.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:10.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:10.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:09 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:58:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:10.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:09 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:58:10.382 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:58:10] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.43.0" 2024-03-12T03:58:12.031 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:11 smithi098 ceph-mon[24773]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-12T03:58:12.031 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:11 smithi098 ceph-mon[29473]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-12T03:58:12.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:11 smithi153 ceph-mon[25643]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 615 B/s rd, 0 op/s 2024-03-12T03:58:12.291 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:58:12.076+0000 7f5f8126c700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:12.292 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:58:12.293 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return future.result(timeout) 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return self.__get_result() 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: raise self._exception 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: config_blobs=daemon_spec.final_config, 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Creating ceph-iscsi config... 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:12.294 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: Traceback (most recent call last): 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: return _run_code(code, main_globals, None, 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:12.295 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: exec(code, run_globals) 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.296 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.777 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:12.777 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:58:12.777 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:12.777 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:12.778 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:58:12.779 INFO:journalctl@ceph.mon.a.smithi098.stdout: return future.result(timeout) 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: return self.__get_result() 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: raise self._exception 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:12.780 INFO:journalctl@ceph.mon.a.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:12.781 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.782 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:58:12.783 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:58:12.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:13.036 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:58:13.037 INFO:journalctl@ceph.mon.c.smithi098.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: return future.result(timeout) 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: return self.__get_result() 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: raise self._exception 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:58:13.038 INFO:journalctl@ceph.mon.c.smithi098.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: Creating ceph-iscsi config... 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: Traceback (most recent call last): 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: exec(code, run_globals) 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:13.039 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.040 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.041 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:58:13.042 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.043 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.044 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:13.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:13.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T03:58:13.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:58:13.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:58:13.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T03:58:13.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:58:13.047 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.048 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:13.049 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:13.050 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:13.050 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:12 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:13.193 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:13.194 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1109, in _check_daemons 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2384, in _daemon_action 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 741, in wait_async 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 64, in get_result 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return future.result(timeout) 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: return self.__get_result() 2024-03-12T03:58:13.195 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: raise self._exception 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1369, in _create_daemon 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: config_blobs=daemon_spec.final_config, 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1671, in _run_cephadm 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Creating ceph-iscsi config... 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Non-zero exit code 1 from systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.196 INFO:journalctl@ceph.mon.b.smithi153.stdout: Traceback (most recent call last): 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: return _run_code(code, main_globals, None, 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: exec(code, run_globals) 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1070, in deploy_daemon 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: RuntimeError: Failed command: systemctl restart ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T03:58:13.197 INFO:journalctl@ceph.mon.b.smithi153.stdout: See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T03:58:13.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:58:13.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.201 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:13.202 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:13.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T03:58:13.203 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:12 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[24773]: Upgrade: Finalizing container_image settings 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[24773]: Upgrade: Complete! 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[24773]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[24773]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 1 op/s 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[24773]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[24773]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[29473]: Upgrade: Finalizing container_image settings 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[29473]: Upgrade: Complete! 2024-03-12T03:58:14.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[29473]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-12T03:58:14.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[29473]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 1 op/s 2024-03-12T03:58:14.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[29473]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:58:14.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:13 smithi098 ceph-mon[29473]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:58:14.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:13 smithi153 ceph-mon[25643]: Upgrade: Finalizing container_image settings 2024-03-12T03:58:14.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:13 smithi153 ceph-mon[25643]: Upgrade: Complete! 2024-03-12T03:58:14.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:13 smithi153 ceph-mon[25643]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 659 B/s rd, 0 op/s 2024-03-12T03:58:14.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:13 smithi153 ceph-mon[25643]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 1 op/s 2024-03-12T03:58:14.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:13 smithi153 ceph-mon[25643]: Reconfiguring iscsi.foo.smithi098.wixybe (dependencies changed)... 2024-03-12T03:58:14.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:13 smithi153 ceph-mon[25643]: Reconfiguring daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T03:58:14.913 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:14 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:14.914 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:14 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:14 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 1 op/s 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1571260166' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 1 op/s 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:16.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:16.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:58:16.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:58:16.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:58:16.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:58:16.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:16 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1571260166' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:58:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 1 op/s 2024-03-12T03:58:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:58:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-12T03:58:16.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:58:16.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-12T03:58:16.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:16 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1571260166' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T03:58:18.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:18 smithi098 ceph-mon[24773]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 1 op/s 2024-03-12T03:58:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:18 smithi098 ceph-mon[29473]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 1 op/s 2024-03-12T03:58:18.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:18 smithi153 ceph-mon[25643]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 1 op/s 2024-03-12T03:58:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[24773]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-12T03:58:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:20.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[29473]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-12T03:58:20.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:20.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:20.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:20.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:19 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:19 smithi153 ceph-mon[25643]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 758 B/s rd, 0 op/s 2024-03-12T03:58:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:19 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:19 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:19 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:19 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:20.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:19 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:20.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:19 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:58:20] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.43.0" 2024-03-12T03:58:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:20 smithi098 ceph-mon[24773]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 236 B/s rd, 0 op/s 2024-03-12T03:58:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:20 smithi098 ceph-mon[24773]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-12T03:58:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:20 smithi098 ceph-mon[24773]: Cluster is now healthy 2024-03-12T03:58:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:20 smithi098 ceph-mon[29473]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 236 B/s rd, 0 op/s 2024-03-12T03:58:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:20 smithi098 ceph-mon[29473]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-12T03:58:21.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:20 smithi098 ceph-mon[29473]: Cluster is now healthy 2024-03-12T03:58:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:20 smithi153 ceph-mon[25643]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 236 B/s rd, 0 op/s 2024-03-12T03:58:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:20 smithi153 ceph-mon[25643]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-12T03:58:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:20 smithi153 ceph-mon[25643]: Cluster is now healthy 2024-03-12T03:58:23.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:22 smithi098 ceph-mon[24773]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 0 op/s 2024-03-12T03:58:23.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:22 smithi098 ceph-mon[29473]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 0 op/s 2024-03-12T03:58:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:22 smithi153 ceph-mon[25643]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 777 B/s rd, 0 op/s 2024-03-12T03:58:25.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:24 smithi098 ceph-mon[24773]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-12T03:58:25.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:24 smithi098 ceph-mon[29473]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-12T03:58:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:24 smithi153 ceph-mon[25643]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 638 B/s rd, 0 op/s 2024-03-12T03:58:27.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:26 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:27.010 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:26 smithi098 ceph-mon[24773]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:58:27.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:26 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:27.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:26 smithi098 ceph-mon[29473]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:58:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:26 smithi153 ceph-mon[25643]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:26 smithi153 ceph-mon[25643]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T03:58:27.377 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-12T03:58:28.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:27 smithi098 ceph-mon[24773]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:28.015 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:27 smithi098 ceph-mon[29473]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:27 smithi153 ceph-mon[25643]: from='client.15756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:29.038 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:28 smithi098 ceph-mon[24773]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:58:29.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:28 smithi098 ceph-mon[29473]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:58:29.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:28 smithi153 ceph-mon[25643]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:58:29.352 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:58:30.001 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:29 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1121655214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:30.001 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:29 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1121655214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:30.026 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-12T03:58:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:29 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1121655214' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:30.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:58:30] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.43.0" 2024-03-12T03:58:31.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:30 smithi098 ceph-mon[24773]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:58:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:30 smithi098 ceph-mon[29473]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:58:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:30 smithi153 ceph-mon[25643]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 912 B/s rd, 0 op/s 2024-03-12T03:58:32.090 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)" 2024-03-12T03:58:32.847 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-12T03:58:32.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:32 smithi098 ceph-mon[24773]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:32.871 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:32 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/909867376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:32.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:32 smithi098 ceph-mon[29473]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:32.872 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:32 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/909867376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:32 smithi153 ceph-mon[25643]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:32 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/909867376' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:34.884 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:58:35.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:34 smithi098 ceph-mon[24773]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:34 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:34 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:34 smithi098 ceph-mon[29473]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:35.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:34 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:35.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:34 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:34 smithi153 ceph-mon[25643]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:34 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:35.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:34 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:35.445 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-12T03:58:36.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:35 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/1092457551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:35 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/1092457551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:36.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:35 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/1092457551' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:36.946 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:36 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:36.947 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:36 smithi098 ceph-mon[24773]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:36.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:36 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:36.947 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:36 smithi098 ceph-mon[29473]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:36 smithi153 ceph-mon[25643]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:36 smithi153 ceph-mon[25643]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:38.873 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:38 smithi098 ceph-mon[24773]: from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:38.873 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:38 smithi098 ceph-mon[24773]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:38.873 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:38 smithi098 ceph-mon[29473]: from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:38.873 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:38 smithi098 ceph-mon[29473]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:38.934 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:58:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:38 smithi153 ceph-mon[25643]: from='client.15780 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:39.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:38 smithi153 ceph-mon[25643]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:39.537 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T03:58:40.476 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:58:40] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-03-12T03:58:41.011 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:40 smithi098 ceph-mon[24773]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:41.011 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:40 smithi098 ceph-mon[29473]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:41.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:40 smithi153 ceph-mon[25643]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:41.450 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:58:41.450 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": null, 2024-03-12T03:58:41.451 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": false, 2024-03-12T03:58:41.451 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "", 2024-03-12T03:58:41.451 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T03:58:41.451 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": null, 2024-03-12T03:58:41.451 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T03:58:41.451 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:58:41.451 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:58:42.091 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T03:58:43.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:42 smithi098 ceph-mon[24773]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:43.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:42 smithi098 ceph-mon[24773]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:42 smithi098 ceph-mon[29473]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:43.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:42 smithi098 ceph-mon[29473]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:42 smithi153 ceph-mon[25643]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:43.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:42 smithi153 ceph-mon[25643]: from='client.15786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:44.248 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T03:58:44.873 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-03-12T03:58:45.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:44 smithi098 ceph-mon[24773]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:45.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:44 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2957253334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:58:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:44 smithi098 ceph-mon[29473]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:45.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:44 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2957253334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:58:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:44 smithi153 ceph-mon[25643]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:44 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2957253334' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:58:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:46 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:46 smithi098 ceph-mon[24773]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:46 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:46 smithi098 ceph-mon[29473]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:46 smithi153 ceph-mon[25643]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:46 smithi153 ceph-mon[25643]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:47 smithi098 ceph-mon[24773]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:47.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:47 smithi098 ceph-mon[29473]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:47 smithi153 ceph-mon[25643]: from='client.15798 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:48.793 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[24773]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mon", "hosts": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[24773]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:49.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:49.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[29473]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mon", "hosts": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[29473]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:49.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:48 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:48 smithi153 ceph-mon[25643]: from='client.15804 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mon", "hosts": "smithi153", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:48 smithi153 ceph-mon[25643]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:48 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:58:49.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:48 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:49.502 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T03:58:50.072 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[24773]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:50.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:50.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:50.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:50.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[29473]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:50.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:50.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:50.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:49 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:49 smithi153 ceph-mon[25643]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:49 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:49 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:58:50.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:49 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:50.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:58:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:58:50] "GET /metrics HTTP/1.1" 200 37998 "" "Prometheus/2.43.0" 2024-03-12T03:58:51.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:50 smithi098 ceph-mon[24773]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:51.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:50 smithi098 ceph-mon[24773]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:50 smithi098 ceph-mon[29473]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:51.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:50 smithi098 ceph-mon[29473]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:50 smithi153 ceph-mon[25643]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:50 smithi153 ceph-mon[25643]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:51.838 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (71s) 33s ago 22m 15.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (52s) 48s ago 22m 51.6M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (37s) 33s ago 22m 50.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (2m) 48s ago 27m 451M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (11m) 33s ago 29m 503M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (29m) 33s ago 29m 74.9M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:58:52.487 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (27m) 48s ago 27m 58.1M 2048M 17.2.0 e1d6a67b021e 61c20497c945 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (27m) 33s ago 27m 60.0M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (115s) 33s ago 23m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (109s) 48s ago 23m 15.4M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (27m) 33s ago 27m 56.4M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (26m) 33s ago 26m 58.6M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (26m) 33s ago 26m 50.5M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (26m) 33s ago 26m 53.0M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (25m) 48s ago 25m 59.4M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (25m) 48s ago 25m 54.9M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (24m) 48s ago 24m 52.8M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (24m) 48s ago 24m 55.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (90s) 48s ago 23m 53.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (22m) 33s ago 22m 76.6M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:58:52.488 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (22m) 48s ago 22m 79.1M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:58:52.879 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:58:52.879 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:58:52.879 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:58:52.880 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-12T03:58:52.881 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:58:52.881 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:58:52.881 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: from='client.25711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T03:58:53.115 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:52 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-12T03:58:53.201 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi153", 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "0/1 daemons upgraded", 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading mon daemons", 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:58:53.202 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:58:53.202 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: from='client.25711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: from='client.25711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:53.203 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:53.204 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T03:58:53.204 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:52 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-12T03:58:53.915 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:53 smithi153 ceph-mon[25643]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:53.916 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:53 smithi153 ceph-mon[25643]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:58:53.916 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:53 smithi153 ceph-mon[25643]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:58:53.916 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:53 smithi153 ceph-mon[25643]: Upgrade: Setting container_image for all mgr 2024-03-12T03:58:53.916 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:53 smithi153 ceph-mon[25643]: Upgrade: It appears safe to stop mon.b 2024-03-12T03:58:53.916 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:53 smithi153 ceph-mon[25643]: from='client.25723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:53.916 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:53 smithi153 ceph-mon[25643]: from='client.? 172.21.15.98:0/2855785653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[24773]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[24773]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[24773]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all mgr 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[24773]: Upgrade: It appears safe to stop mon.b 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[24773]: from='client.25723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2855785653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:54.115 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[29473]: from='client.15816 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:54.116 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[29473]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:58:54.116 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[29473]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:58:54.116 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all mgr 2024-03-12T03:58:54.116 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[29473]: Upgrade: It appears safe to stop mon.b 2024-03-12T03:58:54.116 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[29473]: from='client.25723 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:54.116 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:53 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2855785653' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:58:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: Upgrade: Updating mon.b 2024-03-12T03:58:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:58:55.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:58:55.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:55.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:54 smithi153 ceph-mon[25643]: Deploying daemon mon.b on smithi153 2024-03-12T03:58:55.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:55.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:55.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: Upgrade: Updating mon.b 2024-03-12T03:58:55.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:55.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:58:55.223 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[24773]: Deploying daemon mon.b on smithi153 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: from='client.15834 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: Upgrade: Updating mon.b 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:58:55.224 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:54 smithi098 ceph-mon[29473]: Deploying daemon mon.b on smithi153 2024-03-12T03:58:55.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:55 smithi153 systemd[1]: Stopping Ceph mon.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:55.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b[25626]: 2024-03-12T03:58:55.744+0000 7f125719d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:58:55.880 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b[25626]: 2024-03-12T03:58:55.744+0000 7f125719d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-12T03:58:56.138 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:55 smithi153 podman[67793]: 2024-03-12 03:58:55.878042427 +0000 UTC m=+0.221515905 container died 61c20497c94577252efe7976dbe6a98d11641b1af719337c1f16d6aa4ad08f90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, release=754, version=8, GIT_CLEAN=True, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, architecture=x86_64) 2024-03-12T03:58:56.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:56 smithi153 podman[67793]: 2024-03-12 03:58:56.236027331 +0000 UTC m=+0.579500804 container cleanup 61c20497c94577252efe7976dbe6a98d11641b1af719337c1f16d6aa4ad08f90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, version=8, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754) 2024-03-12T03:58:56.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:56 smithi153 bash[67793]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b 2024-03-12T03:58:56.853 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:56 smithi153 podman[67805]: 2024-03-12 03:58:56.512742852 +0000 UTC m=+0.632768734 container remove 61c20497c94577252efe7976dbe6a98d11641b1af719337c1f16d6aa4ad08f90 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, ceph=True, io.openshift.expose-services=, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public) 2024-03-12T03:58:56.853 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:56 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-12T03:58:56.853 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:56 smithi153 systemd[1]: Stopped Ceph mon.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:58:56.853 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:56 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.b.service: Consumed 17.002s CPU time. 2024-03-12T03:58:57.179 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:56 smithi153 systemd[1]: Starting Ceph mon.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:58:57.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 podman[67941]: 2024-03-12 03:58:56.968831179 +0000 UTC m=+0.019158400 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:58:57.180 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 podman[67941]: 2024-03-12 03:58:57.083762519 +0000 UTC m=+0.134089781 container create 17f05af62f31ea6f9f2218af7d2016d50cea013bf004a7a727bd538ad1e1b6f9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 podman[67941]: 2024-03-12 03:58:57.308494556 +0000 UTC m=+0.358821789 container init 17f05af62f31ea6f9f2218af7d2016d50cea013bf004a7a727bd538ad1e1b6f9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 podman[67941]: 2024-03-12 03:58:57.311479729 +0000 UTC m=+0.361806954 container start 17f05af62f31ea6f9f2218af7d2016d50cea013bf004a7a727bd538ad1e1b6f9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: pidfile_write: ignore empty --pid-file 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: load: jerasure load: lrc 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:58:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Git sha 0 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: DB SUMMARY 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: DB Session ID: 2575WV467I8X96XPDN3D 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2842 Bytes 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2670942 ; 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:58:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.env: 0x55e3817b8b20 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.info_log: 0x55e3831d18e0 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.statistics: (nil) 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.use_fsync: 0 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:58:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.db_log_dir: 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.wal_dir: 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.write_buffer_manager: 0x55e3831a4dc0 2024-03-12T03:58:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.unordered_write: 0 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.row_cache: None 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.wal_filter: None 2024-03-12T03:58:57.447 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.wal_compression: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:58:57.449 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:58:57.450 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_open_files: -1 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Compression algorithms supported: 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kZSTD supported: 0 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:58:57.451 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.merge_operator: 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_filter: None 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e3831d19e0) 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:58:57.452 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_type: 0 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_index_type: 0 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_shortening: 1 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: checksum: 4 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: no_block_cache: 0 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache: 0x55e3831d7090 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_options: 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: capacity : 536870912 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: num_shard_bits : 4 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: strict_capacity_limit : 0 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_cache_compressed: (nil) 2024-03-12T03:58:57.453 INFO:journalctl@ceph.mon.b.smithi153.stdout: persistent_cache: (nil) 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size: 4096 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_size_deviation: 10 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_restart_interval: 16 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: index_block_restart_interval: 1 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: metadata_block_size: 4096 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: partition_filters: 0 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: use_delta_encoding: 1 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: filter_policy: bloomfilter 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: whole_key_filtering: 1 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: verify_compression: 0 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: format_version: 5 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_index_compression: 1 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: block_align: 0 2024-03-12T03:58:57.454 INFO:journalctl@ceph.mon.b.smithi153.stdout: max_auto_readahead_size: 262144 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout: prepopulate_block_cache: 0 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression: NoCompression 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.num_levels: 7 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:58:57.455 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:58:57.456 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:58:57.457 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:58:57.458 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:58:57.459 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.ttl: 2592000 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:58:57.460 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28692, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6fc0310f-caf3-42ba-a28c-b452603ba4dc 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710215937341019, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710215937350105, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1640775, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28697, "largest_seqno": 30252, "table_properties": {"data_size": 1634557, "index_size": 3603, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 15053, "raw_average_key_size": 24, "raw_value_size": 1621628, "raw_average_value_size": 2611, "num_data_blocks": 160, "num_entries": 621, "num_filter_entries": 621, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710215937, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6fc0310f-caf3-42ba-a28c-b452603ba4dc", "db_session_id": "2575WV467I8X96XPDN3D", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710215937350215, "job": 1, "event": "recovery_finished"} 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-12T03:58:57.461 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e3832ca000 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: DB pointer 0x55e3832b4000 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** DB Stats ** 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-12T03:58:57.462 INFO:journalctl@ceph.mon.b.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: L0 1/0 1.56 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 517.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: L6 1/0 8.58 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: Sum 2/0 10.14 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 517.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 517.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** Compaction Stats [default] ** 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 517.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:58:57.463 INFO:journalctl@ceph.mon.b.smithi153.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: Cumulative compaction: 0.00 GB write, 130.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: Interval compaction: 0.00 GB write, 130.38 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: Block cache BinnedLRUCache@0x55e3831d7090#2 capacity: 512.00 MB usage: 5.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: starting mon.b rank 2 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???) e3 preinit fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???).mds e1 new map 2024-03-12T03:58:57.464 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???).mds e1 print_map 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout: e1 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout: legacy client fscid: -1 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout: 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout: No filesystems configured 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-12T03:58:57.465 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T03:58:57.466 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T03:58:57.466 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T03:58:57.466 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 ceph-mon[67960]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-03-12T03:58:57.466 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 bash[67941]: 17f05af62f31ea6f9f2218af7d2016d50cea013bf004a7a727bd538ad1e1b6f9 2024-03-12T03:58:57.466 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:57 smithi153 systemd[1]: Started Ceph mon.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:58:58.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:58.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:58.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:58.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: mon.b calling monitor election 2024-03-12T03:58:58.690 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: mon.c calling monitor election 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: mon.a calling monitor election 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: fsmap 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: mgrmap e41: y(active, since 2m), standbys: x 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: overall HEALTH_OK 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:58.691 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:58:58 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: mon.b calling monitor election 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: mon.c calling monitor election 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: mon.a calling monitor election 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: fsmap 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:58:59.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: mgrmap e41: y(active, since 2m), standbys: x 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: overall HEALTH_OK 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: mon.b calling monitor election 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: mon.c calling monitor election 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: mon.a calling monitor election 2024-03-12T03:58:59.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:58:59.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} 2024-03-12T03:58:59.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: fsmap 2024-03-12T03:58:59.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:58:59.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: mgrmap e41: y(active, since 2m), standbys: x 2024-03-12T03:58:59.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: overall HEALTH_OK 2024-03-12T03:58:59.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:58:59.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:58:58 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:00.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:59:00] "GET /metrics HTTP/1.1" 200 37998 "" "Prometheus/2.43.0" 2024-03-12T03:59:00.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:00 smithi153 ceph-mon[67960]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:01.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:00 smithi098 ceph-mon[29473]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:01.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:00 smithi098 ceph-mon[24773]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:02.756 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:02 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:02.358+0000 7f5f84272700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-12T03:59:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:02 smithi098 ceph-mon[29473]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:02 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:02 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:02 smithi098 ceph-mon[24773]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:02 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:02 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:02 smithi153 ceph-mon[67960]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:02 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:02 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:03 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:03 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:03 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:04.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:03 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:04.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:03 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:04.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:03 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:04.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:03 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:04.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:03 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:04.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:03 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:05.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:04 smithi098 ceph-mon[24773]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:05.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:04 smithi098 ceph-mon[29473]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:05.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:04 smithi153 ceph-mon[67960]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:59:07.082 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:59:07.083 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T03:59:07.084 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.085 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T03:59:07.086 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.087 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.088 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.089 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.090 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.091 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:06 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T03:59:07.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T03:59:07.197 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T03:59:07.198 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.199 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:07.200 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:06 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:08.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Detected new or changed devices on smithi153 2024-03-12T03:59:08.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all crash 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mds 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nfs 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nvmeof 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all node-exporter 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all prometheus 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all alertmanager 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all grafana 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all loki 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all promtail 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Finalizing container_image settings 2024-03-12T03:59:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:07 smithi153 ceph-mon[67960]: Upgrade: Complete! 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Detected new or changed devices on smithi153 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all crash 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all mds 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all nfs 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all nvmeof 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all node-exporter 2024-03-12T03:59:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all prometheus 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all alertmanager 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all grafana 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all loki 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all promtail 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Finalizing container_image settings 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[24773]: Upgrade: Complete! 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Detected new or changed devices on smithi153 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all crash 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all mds 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T03:59:08.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all nfs 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all nvmeof 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all node-exporter 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all prometheus 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all alertmanager 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all grafana 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all loki 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all promtail 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Finalizing container_image settings 2024-03-12T03:59:08.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:07 smithi098 ceph-mon[29473]: Upgrade: Complete! 2024-03-12T03:59:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:08 smithi153 ceph-mon[67960]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:09.225 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:08 smithi098 ceph-mon[24773]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:09.225 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:08 smithi098 ceph-mon[29473]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:09 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:10.220 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:09 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:10.220 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:09 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:10.477 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:59:10] "GET /metrics HTTP/1.1" 200 38004 "" "Prometheus/2.43.0" 2024-03-12T03:59:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:10 smithi153 ceph-mon[67960]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:10 smithi098 ceph-mon[24773]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:11.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:10 smithi098 ceph-mon[29473]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:13.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:12 smithi153 ceph-mon[67960]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:13.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:12 smithi098 ceph-mon[24773]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:13.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:12 smithi098 ceph-mon[29473]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:15.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:14 smithi153 ceph-mon[67960]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:15.233 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:14 smithi098 ceph-mon[24773]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:15.234 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:14 smithi098 ceph-mon[29473]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:16 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:16 smithi153 ceph-mon[67960]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:17.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:16 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:16 smithi098 ceph-mon[24773]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:16 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:16 smithi098 ceph-mon[29473]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:19.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:18 smithi153 ceph-mon[67960]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:19.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:18 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:19.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:18 smithi098 ceph-mon[24773]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:19.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:18 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:18 smithi098 ceph-mon[29473]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:18 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:59:20] "GET /metrics HTTP/1.1" 200 38008 "" "Prometheus/2.43.0" 2024-03-12T03:59:21.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:20 smithi153 ceph-mon[67960]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:20 smithi098 ceph-mon[24773]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:20 smithi098 ceph-mon[29473]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:22 smithi153 ceph-mon[67960]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:23.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:22 smithi098 ceph-mon[24773]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:23.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:22 smithi098 ceph-mon[29473]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:24.208 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T03:59:25.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:24 smithi098 ceph-mon[24773]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:25.099 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:24 smithi098 ceph-mon[24773]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:25.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:24 smithi098 ceph-mon[29473]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:25.099 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:24 smithi098 ceph-mon[29473]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:24 smithi153 ceph-mon[67960]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:24 smithi153 ceph-mon[67960]: from='client.15840 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:26.045 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:59:26.045 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (105s) 66s ago 23m 15.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T03:59:26.045 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (86s) 24s ago 23m 80.7M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T03:59:26.045 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (71s) 66s ago 22m 50.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T03:59:26.045 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (2m) 24s ago 28m 452M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (12m) 66s ago 29m 503M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (29m) 66s ago 29m 74.9M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (29s) 24s ago 28m 19.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (28m) 66s ago 28m 60.0M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (2m) 66s ago 24m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (2m) 24s ago 24m 15.0M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (27m) 66s ago 27m 56.4M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (27m) 66s ago 27m 58.6M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (27m) 66s ago 26m 50.5M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (26m) 66s ago 26m 53.0M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (26m) 24s ago 26m 59.7M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:59:26.046 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (25m) 24s ago 25m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:59:26.047 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (25m) 24s ago 25m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:59:26.047 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (25m) 24s ago 25m 55.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:59:26.047 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (2m) 24s ago 24m 63.6M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T03:59:26.047 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (22m) 66s ago 22m 76.6M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:59:26.047 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (22m) 24s ago 22m 79.6M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:59:26.623 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-03-12T03:59:27.150 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:26 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:27.150 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:26 smithi098 ceph-mon[24773]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:27.150 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:26 smithi098 ceph-mon[24773]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:27.151 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:26 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:27.151 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:26 smithi098 ceph-mon[29473]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:27.151 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:26 smithi098 ceph-mon[29473]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:26 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:26 smithi153 ceph-mon[67960]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:27.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:26 smithi153 ceph-mon[67960]: from='client.15846 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:28.695 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:59:28.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:28 smithi098 ceph-mon[24773]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:28.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:28 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2566196698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:28.978 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:28 smithi098 ceph-mon[29473]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:28.979 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:28 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2566196698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:29.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:28 smithi153 ceph-mon[67960]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:29.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:28 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2566196698' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:29.277 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T03:59:30.486 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:59:30] "GET /metrics HTTP/1.1" 200 38008 "" "Prometheus/2.43.0" 2024-03-12T03:59:31.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:30 smithi098 ceph-mon[24773]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:31.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:30 smithi098 ceph-mon[29473]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:31.156 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:59:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": null, 2024-03-12T03:59:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": false, 2024-03-12T03:59:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "", 2024-03-12T03:59:31.156 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T03:59:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": null, 2024-03-12T03:59:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T03:59:31.157 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:59:31.157 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:59:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:30 smithi153 ceph-mon[67960]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:31.744 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T03:59:33.158 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:32 smithi098 ceph-mon[24773]: from='client.15858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:33.159 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:32 smithi098 ceph-mon[24773]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:33.159 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:32 smithi098 ceph-mon[29473]: from='client.15858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:33.159 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:32 smithi098 ceph-mon[29473]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:33.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:32 smithi153 ceph-mon[67960]: from='client.15858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:33.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:32 smithi153 ceph-mon[67960]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:33.641 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T03:59:33.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:33 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/2891898217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:59:33.983 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:33 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:33.983 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:33 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/2891898217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:59:33.983 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:33 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:34.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:33 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2891898217' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T03:59:34.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:33 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:34.264 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-03-12T03:59:35.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:34 smithi153 ceph-mon[67960]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:35.243 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:34 smithi098 ceph-mon[24773]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:35.243 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:34 smithi098 ceph-mon[29473]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:37.045 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:36 smithi098 ceph-mon[24773]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:37.046 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:36 smithi098 ceph-mon[24773]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:37.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:36 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:37.046 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:36 smithi098 ceph-mon[29473]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:36 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:37.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:36 smithi153 ceph-mon[67960]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:38.058 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:38.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:37 smithi153 ceph-mon[67960]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:38.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:37 smithi153 ceph-mon[67960]: from='client.15876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mon", "hosts": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:37 smithi098 ceph-mon[24773]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:37 smithi098 ceph-mon[24773]: from='client.15876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mon", "hosts": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:37 smithi098 ceph-mon[29473]: from='client.15870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:37 smithi098 ceph-mon[29473]: from='client.15876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "mon", "hosts": "smithi098", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:38.724 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T03:59:39.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[24773]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[24773]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[29473]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[29473]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:39.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:39.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:39.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:39 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:39.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:39 smithi153 ceph-mon[67960]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:39.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:39 smithi153 ceph-mon[67960]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:39.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:39 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:39.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:39 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:39.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:39 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T03:59:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:39 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:40.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:40 smithi098 ceph-mon[24773]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:40.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:40 smithi098 ceph-mon[29473]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:40.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:03:59:40] "GET /metrics HTTP/1.1" 200 38006 "" "Prometheus/2.43.0" 2024-03-12T03:59:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:40 smithi153 ceph-mon[67960]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:41.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[24773]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:41.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:41.147 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[29473]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:41.148 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:41.149 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:41.149 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T03:59:41.149 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:41 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-12T03:59:41.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:41 smithi153 ceph-mon[67960]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:41.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:41 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:41.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:41 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:41.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:41 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:41.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:41 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:41.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:41 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T03:59:41.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:41 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-12T03:59:41.610 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T03:59:42.314 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T03:59:42.314 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (2m) 82s ago 23m 15.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T03:59:42.314 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (102s) 40s ago 23m 80.7M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T03:59:42.314 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (87s) 82s ago 22m 50.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (3m) 40s ago 28m 452M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (12m) 82s ago 30m 503M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (30m) 82s ago 30m 74.9M 2048M 17.2.0 e1d6a67b021e 6c18b4840264 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (45s) 40s ago 28m 19.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (28m) 82s ago 28m 60.0M 2048M 17.2.0 e1d6a67b021e bb746519eeb1 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (2m) 82s ago 24m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (2m) 40s ago 24m 15.0M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (28m) 82s ago 28m 56.4M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (27m) 82s ago 27m 58.6M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (27m) 82s ago 27m 50.5M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (26m) 82s ago 26m 53.0M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (26m) 40s ago 26m 59.7M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T03:59:42.315 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (26m) 40s ago 26m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T03:59:42.316 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (25m) 40s ago 25m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T03:59:42.316 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (25m) 40s ago 25m 55.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T03:59:42.316 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (2m) 40s ago 24m 63.6M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T03:59:42.316 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (23m) 82s ago 23m 76.6M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T03:59:42.316 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (23m) 40s ago 23m 79.6M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T03:59:42.342 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:42 smithi153 ceph-mon[67960]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:59:42.342 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:42 smithi153 ceph-mon[67960]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:59:42.343 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:42 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mgr 2024-03-12T03:59:42.343 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:42 smithi153 ceph-mon[67960]: Upgrade: It appears safe to stop mon.a 2024-03-12T03:59:42.374 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[24773]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:59:42.374 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[24773]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:59:42.374 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[24773]: Upgrade: Setting container_image for all mgr 2024-03-12T03:59:42.374 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[24773]: Upgrade: It appears safe to stop mon.a 2024-03-12T03:59:42.375 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[29473]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T03:59:42.375 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[29473]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T03:59:42.375 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[29473]: Upgrade: Setting container_image for all mgr 2024-03-12T03:59:42.375 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:42 smithi098 ceph-mon[29473]: Upgrade: It appears safe to stop mon.a 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 1 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:59:42.718 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: "mds": {}, 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T03:59:42.719 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:59:43.059 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi098", 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "0/2 daemons upgraded", 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading mon daemons", 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T03:59:43.060 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:43.314 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[24773]: from='client.? 172.21.15.98:0/3185904210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:43.315 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:43 smithi098 ceph-mon[29473]: from='client.? 172.21.15.98:0/3185904210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: from='client.15882 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: from='client.15888 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:43.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='mgr.y' 2024-03-12T03:59:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T03:59:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T03:59:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T03:59:43.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:43 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/3185904210' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T03:59:44.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[29473]: from='client.15894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:44.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[29473]: Upgrade: Updating mon.a 2024-03-12T03:59:44.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[29473]: Deploying daemon mon.a on smithi098 2024-03-12T03:59:44.309 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[29473]: from='client.25807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:44.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[24773]: from='client.15894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:44.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[24773]: Upgrade: Updating mon.a 2024-03-12T03:59:44.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[24773]: Deploying daemon mon.a on smithi098 2024-03-12T03:59:44.310 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-mon[24773]: from='client.25807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:44.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:44 smithi153 ceph-mon[67960]: from='client.15894 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:44.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:44 smithi153 ceph-mon[67960]: Upgrade: Updating mon.a 2024-03-12T03:59:44.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:44 smithi153 ceph-mon[67960]: Deploying daemon mon.a on smithi098 2024-03-12T03:59:44.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:44 smithi153 ceph-mon[67960]: from='client.25807 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T03:59:44.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 systemd[1]: Stopping Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:44.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a[24769]: 2024-03-12T03:59:44.570+0000 7fa32a071700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T03:59:44.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a[24769]: 2024-03-12T03:59:44.570+0000 7fa32a071700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-12T03:59:44.965 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:44 smithi098 podman[85859]: 2024-03-12 03:59:44.66264308 +0000 UTC m=+0.212244103 container died 6c18b4840264544c5e2ddba53391c208e5af879c87c6d72e4e608288272d060d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, version=8, architecture=x86_64, name=centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-type=git, ceph=True) 2024-03-12T03:59:45.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 podman[85859]: 2024-03-12 03:59:45.06269291 +0000 UTC m=+0.612293941 container cleanup 6c18b4840264544c5e2ddba53391c208e5af879c87c6d72e4e608288272d060d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_BRANCH=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-12T03:59:45.253 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 bash[85859]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a 2024-03-12T03:59:45.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 podman[85873]: 2024-03-12 03:59:45.305554581 +0000 UTC m=+0.647363765 container remove 6c18b4840264544c5e2ddba53391c208e5af879c87c6d72e4e608288272d060d (image=quay.io/ceph/ceph:v17.2.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, ceph=True, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, version=8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc.) 2024-03-12T03:59:45.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-12T03:59:45.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 systemd[1]: Stopped Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:59:45.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service: Consumed 25.430s CPU time. 2024-03-12T03:59:45.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 systemd[1]: Starting Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T03:59:45.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 podman[86002]: 2024-03-12 03:59:45.772290912 +0000 UTC m=+0.018449350 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T03:59:45.985 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:45 smithi098 podman[86002]: 2024-03-12 03:59:45.881953844 +0000 UTC m=+0.128112264 container create ee9742767c6257d6d6969b013cf4f0b9bf026630889ad1143d35396980568408 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T03:59:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 podman[86002]: 2024-03-12 03:59:46.118294644 +0000 UTC m=+0.364453083 container init ee9742767c6257d6d6969b013cf4f0b9bf026630889ad1143d35396980568408 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-12T03:59:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 podman[86002]: 2024-03-12 03:59:46.121587046 +0000 UTC m=+0.367745480 container start ee9742767c6257d6d6969b013cf4f0b9bf026630889ad1143d35396980568408 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T03:59:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T03:59:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T03:59:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: pidfile_write: ignore empty --pid-file 2024-03-12T03:59:46.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: load: jerasure load: lrc 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: RocksDB version: 7.9.2 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Git sha 0 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: DB SUMMARY 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: DB Session ID: 3SM7LQV8CUOCY8XBLA85 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: CURRENT file: CURRENT 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: MANIFEST file: MANIFEST-000015 size: 3108 Bytes 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000063.sst 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000061.log size: 92163 ; 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.error_if_exists: 0 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.create_if_missing: 0 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.paranoid_checks: 1 2024-03-12T03:59:46.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.env: 0x560a4a5beb20 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.info_log: 0x560a4c0d98e0 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.statistics: (nil) 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.use_fsync: 0 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_log_file_size: 0 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T03:59:46.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.allow_fallocate: 1 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.use_direct_reads: 0 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.db_log_dir: 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.wal_dir: 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T03:59:46.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.write_buffer_manager: 0x560a4c0acdc0 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.unordered_write: 0 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T03:59:46.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T03:59:46.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T03:59:46.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.row_cache: None 2024-03-12T03:59:46.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.wal_filter: None 2024-03-12T03:59:46.329 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T03:59:46.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T03:59:46.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.two_write_queues: 0 2024-03-12T03:59:46.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T03:59:46.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.wal_compression: 0 2024-03-12T03:59:46.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.atomic_flush: 0 2024-03-12T03:59:46.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T03:59:46.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.log_readahead_size: 0 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_background_jobs: 2 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_background_compactions: -1 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_subcompactions: 1 2024-03-12T03:59:46.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_open_files: -1 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T03:59:46.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_background_flushes: -1 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Compression algorithms supported: 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kZSTD supported: 0 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kXpressCompression supported: 0 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kZlibCompression supported: 1 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kSnappyCompression supported: 1 2024-03-12T03:59:46.333 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kLZ4Compression supported: 1 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: kBZip2Compression supported: 0 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.merge_operator: 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_filter: None 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T03:59:46.334 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560a4c0d99e0) 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_type: 0 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_index_type: 0 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_shortening: 1 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: checksum: 4 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: no_block_cache: 0 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache: 0x560a4c0df090 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-03-12T03:59:46.335 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_options: 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: capacity : 536870912 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_shard_bits : 4 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: strict_capacity_limit : 0 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_cache_compressed: (nil) 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: persistent_cache: (nil) 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size: 4096 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_size_deviation: 10 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_restart_interval: 16 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: index_block_restart_interval: 1 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: metadata_block_size: 4096 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: partition_filters: 0 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: use_delta_encoding: 1 2024-03-12T03:59:46.336 INFO:journalctl@ceph.mon.a.smithi098.stdout: filter_policy: bloomfilter 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: whole_key_filtering: 1 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: verify_compression: 0 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: format_version: 5 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_index_compression: 1 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: block_align: 0 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: max_auto_readahead_size: 262144 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: prepopulate_block_cache: 0 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: initial_auto_readahead_size: 8192 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T03:59:46.337 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression: NoCompression 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.num_levels: 7 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T03:59:46.338 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T03:59:46.339 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T03:59:46.340 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T03:59:46.341 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.table_properties_collectors: 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.inplace_update_support: 0 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.bloom_locality: 0 2024-03-12T03:59:46.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.max_successive_merges: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.ttl: 2592000 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.enable_blob_files: false 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.min_blob_size: 0 2024-03-12T03:59:46.343 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 63.sst 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 65, last_sequence is 29256, log_number is 61,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 61 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: cde4c51e-7bca-4777-9103-737a42b044e0 2024-03-12T03:59:46.344 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710215986162027, "job": 1, "event": "recovery_started", "wal_files": [61]} 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #61 mode 2 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710215986162811, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 66, "file_size": 91946, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 29257, "largest_seqno": 29370, "table_properties": {"data_size": 90455, "index_size": 287, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 1821, "raw_average_key_size": 26, "raw_value_size": 88902, "raw_average_value_size": 1270, "num_data_blocks": 13, "num_entries": 70, "num_filter_entries": 70, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710215986, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "cde4c51e-7bca-4777-9103-737a42b044e0", "db_session_id": "3SM7LQV8CUOCY8XBLA85", "orig_file_number": 66, "seqno_to_time_mapping": "N/A"}} 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710215986162909, "job": 1, "event": "recovery_finished"} 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/version_set.cc:5047] Creating manifest 68 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000061.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560a4c1d2000 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: DB pointer 0x560a4c1bc000 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T03:59:46.345 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** DB Stats ** 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T03:59:46.346 INFO:journalctl@ceph.mon.a.smithi098.stdout: L0 1/0 89.79 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: L6 1/0 9.27 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: Sum 2/0 9.36 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:59:46.347 INFO:journalctl@ceph.mon.a.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: Cumulative compaction: 0.00 GB write, 6.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: Interval compaction: 0.00 GB write, 6.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache BinnedLRUCache@0x560a4c0df090#2 capacity: 512.00 MB usage: 0.59 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.36 KB,6.85453e-05%) Misc(1,0.00 KB,0%) 2024-03-12T03:59:46.348 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: starting mon.a rank 0 at public addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] at bind addrs [v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???) e3 preinit fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???).mds e1 new map 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???).mds e1 print_map 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout: e1 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout: legacy client fscid: -1 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout: 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout: No filesystems configured 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T03:59:46.349 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T03:59:46.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T03:59:46.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 ceph-mon[86021]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-03-12T03:59:46.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 bash[86002]: ee9742767c6257d6d6969b013cf4f0b9bf026630889ad1143d35396980568408 2024-03-12T03:59:46.350 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:46 smithi098 systemd[1]: Started Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T03:59:49.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ignoring --setuser ceph since I am not root 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ignoring --setgroup ceph since I am not root 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mgr[63028]: -- 172.21.15.98:0/636151626 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570ef8685a0 con 0x5570ef83c800 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: mon.a calling monitor election 2024-03-12T03:59:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: fsmap 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: overall HEALTH_OK 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: from='mgr.25249 ' entity='' 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[86021]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:49.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: mon.a calling monitor election 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: fsmap 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: overall HEALTH_OK 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: from='mgr.25249 ' entity='' 2024-03-12T03:59:49.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-mon[29473]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-12T03:59:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: from='mgr.25249 172.21.15.98:0/1694628992' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: mon.a calling monitor election 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: monmap e3: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: fsmap 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: osdmap e112: 8 total, 8 up, 8 in 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: overall HEALTH_OK 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: from='mgr.25249 ' entity='' 2024-03-12T03:59:49.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mon[67960]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-12T03:59:49.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: ignoring --setuser ceph since I am not root 2024-03-12T03:59:49.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: ignoring --setgroup ceph since I am not root 2024-03-12T03:59:49.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-mgr[61139]: -- 172.21.15.153:0/1361522496 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563e20c7c5a0 con 0x563e20c50800 2024-03-12T03:59:50.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:49.793+0000 7f7a1895c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:59:50.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:49 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:49.886+0000 7f7a1895c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:59:50.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:49.795+0000 7f6b81959200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-12T03:59:50.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:49 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:49.889+0000 7f6b81959200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-12T03:59:50.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:50.524+0000 7f7a1895c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:59:50.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:50.616+0000 7f7a1895c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:59:50.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:50.757+0000 7f7a1895c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:59:50.861 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:50.528+0000 7f6b81959200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-12T03:59:50.861 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:50.620+0000 7f6b81959200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-12T03:59:50.861 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:50.763+0000 7f6b81959200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-12T03:59:50.861 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:50.859+0000 7f6b81959200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:59:51.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:50.943+0000 7f6b81959200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:59:51.206 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:50.851+0000 7f7a1895c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-12T03:59:51.206 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:50.934+0000 7f7a1895c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-12T03:59:51.500 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:51 smithi098 ceph-mon[86021]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-12T03:59:51.500 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:51 smithi098 ceph-mon[29473]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-12T03:59:51.500 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:51.203+0000 7f7a1895c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:59:51.500 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:51.316+0000 7f7a1895c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:59:51.500 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:51.402+0000 7f7a1895c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:59:51.500 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:51.497+0000 7f7a1895c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:59:51.506 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:51 smithi153 ceph-mon[67960]: mgrmap e43: y(active, since 3m), standbys: x 2024-03-12T03:59:51.506 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:51 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:51.210+0000 7f6b81959200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-12T03:59:51.506 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:51 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:51.322+0000 7f6b81959200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-12T03:59:51.507 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:51 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:51.409+0000 7f6b81959200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-12T03:59:51.507 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:51 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:51.504+0000 7f6b81959200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-12T03:59:51.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:51.674+0000 7f7a1895c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:59:51.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:51 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:51.682+0000 7f6b81959200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-12T03:59:52.774 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:52 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:52.642+0000 7f7a1895c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:59:52.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:52.646+0000 7f6b81959200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-12T03:59:52.942 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:52.776+0000 7f6b81959200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:59:53.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:52 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:52.772+0000 7f7a1895c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-12T03:59:53.398 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:53.106+0000 7f7a1895c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:59:53.399 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:53.198+0000 7f7a1895c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:59:53.404 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:53.113+0000 7f6b81959200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-12T03:59:53.404 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:53.205+0000 7f6b81959200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-12T03:59:53.404 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:53.402+0000 7f6b81959200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:59:53.694 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:53.594+0000 7f6b81959200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:59:53.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:53.396+0000 7f7a1895c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:59:53.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:53.589+0000 7f7a1895c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-12T03:59:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:59:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:59:53.514Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T03:59:53.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:59:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:59:53.515Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T03:59:54.144 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:53 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:53.882+0000 7f7a1895c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:59:54.150 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:53.887+0000 7f6b81959200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-12T03:59:54.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:54 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:54.149+0000 7f6b81959200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:59:54.442 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:54 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:54.391+0000 7f6b81959200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:59:54.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:54 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:54.141+0000 7f7a1895c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-12T03:59:54.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:54 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:54.382+0000 7f7a1895c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-12T03:59:55.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:55.196+0000 7f7a1895c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:59:55.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:55.203+0000 7f6b81959200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-12T03:59:56.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:55.773+0000 7f7a1895c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:59:56.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:55.876+0000 7f7a1895c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:59:56.108 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:55.785+0000 7f6b81959200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-12T03:59:56.108 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:55 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:55.886+0000 7f6b81959200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-12T03:59:56.108 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:56.106+0000 7f6b81959200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:59:56.359 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:56.101+0000 7f7a1895c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-12T03:59:56.359 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:59:56] ENGINE Bus STARTING 2024-03-12T03:59:56.359 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: Active manager daemon y restarted 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: Activating manager daemon y 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: mgrmap e44: y(active, starting, since 0.00621263s), standbys: x 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: Standby manager daemon x restarted 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: Standby manager daemon x started 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:59:56.360 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:59:56.361 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[29473]: Manager daemon y is now available 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: Active manager daemon y restarted 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: Activating manager daemon y 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: mgrmap e44: y(active, starting, since 0.00621263s), standbys: x 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: Standby manager daemon x restarted 2024-03-12T03:59:56.362 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: Standby manager daemon x started 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:59:56.363 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:59:56.364 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-mon[86021]: Manager daemon y is now available 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: [12/Mar/2024:03:59:56] ENGINE Bus STARTING 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: CherryPy Checker: 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: The Application mounted at '' has an empty config. 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: [12/Mar/2024:03:59:56] ENGINE Serving on http://:::9283 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x[61135]: [12/Mar/2024:03:59:56] ENGINE Bus STARTED 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: Active manager daemon y restarted 2024-03-12T03:59:56.366 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: Activating manager daemon y 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: mgrmap e44: y(active, starting, since 0.00621263s), standbys: x 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: Standby manager daemon x restarted 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: Standby manager daemon x started 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.? 172.21.15.153:0/1763648993' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-12T03:59:56.367 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-12T03:59:56.368 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:56 smithi153 ceph-mon[67960]: Manager daemon y is now available 2024-03-12T03:59:56.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CherryPy Checker: 2024-03-12T03:59:56.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: The Application mounted at '' has an empty config. 2024-03-12T03:59:56.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:56.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:59:56] ENGINE Serving on http://:::9283 2024-03-12T03:59:56.824 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:03:59:56] ENGINE Bus STARTED 2024-03-12T03:59:57.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:57.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:57.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:59:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:57 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:59:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:59:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:57 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:59:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:57 smithi153 ceph-mon[67960]: mgrmap e45: y(active, since 1.00948s), standbys: x 2024-03-12T03:59:57.472 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:59:57.208Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=5 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T03:59:57.127+0000 7f79ceb0e700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:59:57.473 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[86021]: mgrmap e45: y(active, since 1.00948s), standbys: x 2024-03-12T03:59:57.474 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T03:59:57.474 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T03:59:57.474 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:59:57.474 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-12T03:59:57.474 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:59:57.474 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-12T03:59:57.474 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-mon[29473]: mgrmap e45: y(active, since 1.00948s), standbys: x 2024-03-12T03:59:58.171 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 03:59:57 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T03:59:57.869Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=5 2024-03-12T03:59:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:58 smithi153 ceph-mon[67960]: [12/Mar/2024:03:59:56] ENGINE Bus STARTING 2024-03-12T03:59:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:58 smithi153 ceph-mon[67960]: [12/Mar/2024:03:59:56] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:59:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:58 smithi153 ceph-mon[67960]: [12/Mar/2024:03:59:56] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:59:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:58 smithi153 ceph-mon[67960]: [12/Mar/2024:03:59:56] ENGINE Bus STARTED 2024-03-12T03:59:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:58 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:58 smithi153 ceph-mon[67960]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:59:58.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[86021]: [12/Mar/2024:03:59:56] ENGINE Bus STARTING 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[86021]: [12/Mar/2024:03:59:56] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[86021]: [12/Mar/2024:03:59:56] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[86021]: [12/Mar/2024:03:59:56] ENGINE Bus STARTED 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[86021]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[29473]: [12/Mar/2024:03:59:56] ENGINE Bus STARTING 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[29473]: [12/Mar/2024:03:59:56] ENGINE Serving on http://172.21.15.98:8765 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[29473]: [12/Mar/2024:03:59:56] ENGINE Serving on https://172.21.15.98:7150 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[29473]: [12/Mar/2024:03:59:56] ENGINE Bus STARTED 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T03:59:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:58 smithi098 ceph-mon[29473]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:59:59.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:59 smithi153 ceph-mon[67960]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:59:59.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 03:59:59 smithi153 ceph-mon[67960]: mgrmap e46: y(active, since 3s), standbys: x 2024-03-12T03:59:59.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:59 smithi098 ceph-mon[86021]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:59:59.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 03:59:59 smithi098 ceph-mon[86021]: mgrmap e46: y(active, since 3s), standbys: x 2024-03-12T03:59:59.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:59 smithi098 ceph-mon[29473]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T03:59:59.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 03:59:59 smithi098 ceph-mon[29473]: mgrmap e46: y(active, since 3s), standbys: x 2024-03-12T04:00:00.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:00 smithi153 ceph-mon[67960]: overall HEALTH_OK 2024-03-12T04:00:00.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:00 smithi098 ceph-mon[86021]: overall HEALTH_OK 2024-03-12T04:00:00.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:00 smithi098 ceph-mon[29473]: overall HEALTH_OK 2024-03-12T04:00:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:00:00] "GET /metrics HTTP/1.1" 200 35205 "" "Prometheus/2.43.0" 2024-03-12T04:00:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:01 smithi153 ceph-mon[67960]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:00:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:01 smithi153 ceph-mon[67960]: mgrmap e47: y(active, since 4s), standbys: x 2024-03-12T04:00:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:01 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:01 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:01 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:01 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[86021]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:00:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[86021]: mgrmap e47: y(active, since 4s), standbys: x 2024-03-12T04:00:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[29473]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[29473]: mgrmap e47: y(active, since 4s), standbys: x 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:01 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:02.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:02 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:02.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:02 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:02.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:02 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:02.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:02 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:02.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:02 smithi153 ceph-mon[67960]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-12T04:00:03.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[86021]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:03.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:02 smithi098 ceph-mon[29473]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-12T04:00:05.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:05 smithi153 ceph-mon[67960]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T04:00:05.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:05 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:05.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:05 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:05.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:05 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:05.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:05 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[86021]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T04:00:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:05.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:05.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[29473]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-12T04:00:05.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:05.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:05 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:06 smithi153 ceph-mon[67960]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T04:00:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:06 smithi153 ceph-mon[67960]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T04:00:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:06 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:06 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:06 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:06 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:06 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[86021]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T04:00:06.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[86021]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T04:00:06.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.conf 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.conf 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:06.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:06 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:07.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[86021]: Upgrade: It appears safe to stop mon.c 2024-03-12T04:00:07.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-12T04:00:07.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:07 smithi098 ceph-mon[29473]: Upgrade: It appears safe to stop mon.c 2024-03-12T04:00:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T04:00:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-12T04:00:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: Updating smithi098:/etc/ceph/ceph.client.admin.keyring 2024-03-12T04:00:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-03-12T04:00:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: Updating smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T04:00:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: Updating smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-12T04:00:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-12T04:00:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:00:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-12T04:00:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-12T04:00:07.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:07 smithi153 ceph-mon[67960]: Upgrade: It appears safe to stop mon.c 2024-03-12T04:00:08.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:08.452 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[86021]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[29473]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[29473]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:08.453 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:08 smithi098 ceph-mon[29473]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:08 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:08 smithi153 ceph-mon[67960]: from='mgr.25819 ' entity='mgr.y' 2024-03-12T04:00:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:08.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:09.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-mon[86021]: Upgrade: Updating mon.c 2024-03-12T04:00:09.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-mon[86021]: Deploying daemon mon.c on smithi098 2024-03-12T04:00:09.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-mon[86021]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T04:00:09.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-mon[29473]: Upgrade: Updating mon.c 2024-03-12T04:00:09.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-mon[29473]: Deploying daemon mon.c on smithi098 2024-03-12T04:00:09.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-mon[29473]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T04:00:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:09 smithi153 ceph-mon[67960]: Upgrade: Updating mon.c 2024-03-12T04:00:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:09 smithi153 ceph-mon[67960]: Deploying daemon mon.c on smithi098 2024-03-12T04:00:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:09 smithi153 ceph-mon[67960]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-12T04:00:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:09 smithi098 systemd[1]: Stopping Ceph mon.c for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:00:10.221 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c[29469]: 2024-03-12T04:00:09.821+0000 7fb2145d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:00:10.221 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c[29469]: 2024-03-12T04:00:09.821+0000 7fb2145d5700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-12T04:00:10.221 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:09 smithi098 podman[88228]: 2024-03-12 04:00:09.966186241 +0000 UTC m=+0.244748244 container died bb746519eeb151cd78a1bc0bf4ea8f145247a6de0cd3bd594c3199f05e644858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, RELEASE=HEAD, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-type=git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, ceph=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, GIT_CLEAN=True) 2024-03-12T04:00:10.222 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:00:09] ENGINE Bus STOPPING 2024-03-12T04:00:10.222 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:00:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T04:00:10.222 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:00:09] ENGINE Bus STOPPED 2024-03-12T04:00:10.222 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:00:09] ENGINE Bus STARTING 2024-03-12T04:00:10.222 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:00:09] ENGINE Serving on http://:::9283 2024-03-12T04:00:10.222 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:09 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:00:09] ENGINE Bus STARTED 2024-03-12T04:00:10.222 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:00:10] "GET /metrics HTTP/1.1" 200 35205 "" "Prometheus/2.43.0" 2024-03-12T04:00:10.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:10 smithi098 podman[88228]: 2024-03-12 04:00:10.314171948 +0000 UTC m=+0.592734002 container cleanup bb746519eeb151cd78a1bc0bf4ea8f145247a6de0cd3bd594c3199f05e644858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vcs-type=git, release=754, architecture=x86_64, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, version=8, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-12T04:00:10.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:10 smithi098 bash[88228]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c 2024-03-12T04:00:10.935 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:10 smithi098 podman[88253]: 2024-03-12 04:00:10.57368699 +0000 UTC m=+0.603671259 container remove bb746519eeb151cd78a1bc0bf4ea8f145247a6de0cd3bd594c3199f05e644858 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, name=centos-stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.component=centos-stream-container, release=754, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-12T04:00:10.936 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:10 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-12T04:00:10.936 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:10 smithi098 systemd[1]: Stopped Ceph mon.c for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:00:10.936 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:10 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.c.service: Consumed 15.991s CPU time. 2024-03-12T04:00:11.281 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:10 smithi098 systemd[1]: Starting Ceph mon.c for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:00:11.281 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 podman[88387]: 2024-03-12 04:00:11.053524178 +0000 UTC m=+0.017214155 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:00:11.281 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 podman[88387]: 2024-03-12 04:00:11.158053265 +0000 UTC m=+0.121743236 container create 3a3af35df14b6e7b117bddafc1582095543bf312c07e81ee63e46942ad8570cc (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T04:00:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 podman[88387]: 2024-03-12 04:00:11.402365977 +0000 UTC m=+0.366055957 container init 3a3af35df14b6e7b117bddafc1582095543bf312c07e81ee63e46942ad8570cc (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-12T04:00:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 podman[88387]: 2024-03-12 04:00:11.406115586 +0000 UTC m=+0.369805560 container start 3a3af35df14b6e7b117bddafc1582095543bf312c07e81ee63e46942ad8570cc (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T04:00:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: set uid:gid to 167:167 (ceph:ceph) 2024-03-12T04:00:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-12T04:00:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: pidfile_write: ignore empty --pid-file 2024-03-12T04:00:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: load: jerasure load: lrc 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: RocksDB version: 7.9.2 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Git sha 0 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: DB SUMMARY 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: DB Session ID: IJTDVQ8DF621XZBVHGLU 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: CURRENT file: CURRENT 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: IDENTITY file: IDENTITY 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3223 Bytes 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2308185 ; 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.error_if_exists: 0 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.create_if_missing: 0 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.paranoid_checks: 1 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-12T04:00:11.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-12T04:00:11.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.env: 0x560d9682cb20 2024-03-12T04:00:11.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.fs: PosixFileSystem 2024-03-12T04:00:11.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.info_log: 0x560d98e6f8e0 2024-03-12T04:00:11.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_file_opening_threads: 16 2024-03-12T04:00:11.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.statistics: (nil) 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.use_fsync: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_log_file_size: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.keep_log_file_num: 1000 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.recycle_log_file_num: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.allow_fallocate: 1 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.allow_mmap_reads: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.allow_mmap_writes: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.use_direct_reads: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.create_missing_column_families: 0 2024-03-12T04:00:11.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.db_log_dir: 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.wal_dir: 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.advise_random_on_open: 1 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.db_write_buffer_size: 0 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.write_buffer_manager: 0x560d98e42dc0 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-12T04:00:11.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.rate_limiter: (nil) 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.wal_recovery_mode: 2 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.enable_thread_tracking: 0 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.enable_pipelined_write: 0 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.unordered_write: 0 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.row_cache: None 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.wal_filter: None 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.allow_ingest_behind: 0 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.two_write_queues: 0 2024-03-12T04:00:11.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.manual_wal_flush: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.wal_compression: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.atomic_flush: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.log_readahead_size: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.best_efforts_recovery: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.allow_data_in_errors: 0 2024-03-12T04:00:11.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.db_host_id: __hostname__ 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_background_jobs: 2 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_background_compactions: -1 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_subcompactions: 1 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_total_wal_size: 0 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_open_files: -1 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bytes_per_sync: 0 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-12T04:00:11.581 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_readahead_size: 0 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_background_flushes: -1 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Compression algorithms supported: 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kZSTD supported: 0 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kXpressCompression supported: 0 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kLZ4HCCompression supported: 1 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kZlibCompression supported: 1 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kSnappyCompression supported: 1 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kLZ4Compression supported: 1 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: kBZip2Compression supported: 0 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-12T04:00:11.582 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.merge_operator: 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_filter: None 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_filter_factory: None 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.sst_partitioner_factory: None 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560d98e6f9e0) 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks: 1 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout: pin_top_level_index_and_filter: 1 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_type: 0 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_index_type: 0 2024-03-12T04:00:11.583 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_shortening: 1 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: checksum: 4 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: no_block_cache: 0 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache: 0x560d98e75090 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_name: BinnedLRUCache 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_options: 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: capacity : 536870912 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: num_shard_bits : 4 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: strict_capacity_limit : 0 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: high_pri_pool_ratio: 0.000 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_cache_compressed: (nil) 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: persistent_cache: (nil) 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size: 4096 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_size_deviation: 10 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_restart_interval: 16 2024-03-12T04:00:11.584 INFO:journalctl@ceph.mon.c.smithi098.stdout: index_block_restart_interval: 1 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: metadata_block_size: 4096 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: partition_filters: 0 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: use_delta_encoding: 1 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: filter_policy: bloomfilter 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: whole_key_filtering: 1 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: verify_compression: 0 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: read_amp_bytes_per_bit: 0 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: format_version: 5 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_index_compression: 1 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: block_align: 0 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: max_auto_readahead_size: 262144 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: prepopulate_block_cache: 0 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: initial_auto_readahead_size: 8192 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout: num_file_reads_for_auto_readahead: 2 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.write_buffer_size: 33554432 2024-03-12T04:00:11.585 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_write_buffer_number: 2 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression: NoCompression 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression: Disabled 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.prefix_extractor: nullptr 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.num_levels: 7 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-12T04:00:11.586 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.level: 32767 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.strategy: 0 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.enabled: false 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.target_file_size_base: 67108864 2024-03-12T04:00:11.587 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-12T04:00:11.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-12T04:00:11.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-12T04:00:11.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-12T04:00:11.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-12T04:00:11.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-12T04:00:11.588 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.arena_block_size: 1048576 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.disable_auto_compactions: 0 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-12T04:00:11.589 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.table_properties_collectors: 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.inplace_update_support: 0 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-12T04:00:11.590 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.bloom_locality: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.max_successive_merges: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.paranoid_file_checks: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.force_consistency_checks: 1 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.report_bg_io_stats: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.ttl: 2592000 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.enable_blob_files: false 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.min_blob_size: 0 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.blob_file_size: 268435456 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-12T04:00:11.591 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.blob_file_starting_level: 0 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32159, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f5a2bf00-bde9-4c54-a32f-d016e0a3bf5c 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710216011435518, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710216011441158, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1470297, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32164, "largest_seqno": 32568, "table_properties": {"data_size": 1467428, "index_size": 1343, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4712, "raw_average_key_size": 24, "raw_value_size": 1463010, "raw_average_value_size": 7502, "num_data_blocks": 60, "num_entries": 195, "num_filter_entries": 195, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710216011, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f5a2bf00-bde9-4c54-a32f-d016e0a3bf5c", "db_session_id": "IJTDVQ8DF621XZBVHGLU", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710216011441258, "job": 1, "event": "recovery_finished"} 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-12T04:00:11.592 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560d98f68000 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: DB pointer 0x560d98f52000 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** DB Stats ** 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T04:00:11.593 INFO:journalctl@ceph.mon.c.smithi098.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: L0 1/0 1.40 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 550.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: L6 1/0 9.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Sum 2/0 11.16 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 550.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 550.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** Compaction Stats [default] ** 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 550.3 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T04:00:11.594 INFO:journalctl@ceph.mon.c.smithi098.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Cumulative compaction: 0.00 GB write, 138.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Interval compaction: 0.00 GB write, 138.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Block cache BinnedLRUCache@0x560d98e75090#2 capacity: 512.00 MB usage: 2.05 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.45 KB,0.000277162%) Misc(1,0.00 KB,0%) 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: starting mon.c rank 1 at public addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] at bind addrs [v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???) e3 preinit fsid ac0aaff8-e020-11ee-95c8-87774f69a715 2024-03-12T04:00:11.595 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???).mds e1 new map 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???).mds e1 print_map 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: e1 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: legacy client fscid: -1 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout: No filesystems configured 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???).osd e113 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???).osd e113 crush map has features 432629239337189376, adjusting msgr requires 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 ceph-mon[88407]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 bash[88387]: 3a3af35df14b6e7b117bddafc1582095543bf312c07e81ee63e46942ad8570cc 2024-03-12T04:00:11.596 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:11 smithi098 systemd[1]: Started Ceph mon.c for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: mon.a calling monitor election 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: mon.c calling monitor election 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: mon.b calling monitor election 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: monmap e4: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: fsmap 2024-03-12T04:00:12.741 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: mgrmap e47: y(active, since 15s), standbys: x 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: overall HEALTH_OK 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[86021]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: mon.a calling monitor election 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: mon.c calling monitor election 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: mon.b calling monitor election 2024-03-12T04:00:12.742 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: monmap e4: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: fsmap 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: mgrmap e47: y(active, since 15s), standbys: x 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: overall HEALTH_OK 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:12.743 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:12 smithi098 ceph-mon[88407]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T04:00:12.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-12T04:00:12.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: mon.a calling monitor election 2024-03-12T04:00:12.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: mon.c calling monitor election 2024-03-12T04:00:12.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: mon.b calling monitor election 2024-03-12T04:00:12.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: monmap e4: 3 mons at {a=[v2:172.21.15.98:3300/0,v1:172.21.15.98:6789/0],b=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],c=[v2:172.21.15.98:3301/0,v1:172.21.15.98:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: fsmap 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: osdmap e113: 8 total, 8 up, 8 in 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: mgrmap e47: y(active, since 15s), standbys: x 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: overall HEALTH_OK 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:12.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:12 smithi153 ceph-mon[67960]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T04:00:13.414 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:00:14.118 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (2m) 13s ago 24m 28.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (2m) 13s ago 24m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (119s) 13s ago 23m 50.9M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (3m) 13s ago 28m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (13m) 13s ago 30m 474M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (28s) 13s ago 30m 30.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (77s) 13s ago 29m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 starting - - - 2048M 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (3m) 13s ago 25m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (3m) 13s ago 25m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (28m) 13s ago 28m 56.6M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (28m) 13s ago 28m 58.7M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T04:00:14.119 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (27m) 13s ago 27m 50.7M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (27m) 13s ago 27m 53.3M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (27m) 13s ago 27m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (26m) 13s ago 26m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (26m) 13s ago 26m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (25m) 13s ago 25m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (2m) 13s ago 25m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (23m) 13s ago 23m 77.3M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:00:14.120 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (23m) 13s ago 23m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:00:14.517 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:00:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:00:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:00:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:00:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:00:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:00:14.517 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 5 2024-03-12T04:00:14.518 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:00:14.519 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi098", 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "1/2 daemons upgraded", 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading mon daemons", 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:00:14.859 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:00:15.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:15 smithi153 ceph-mon[67960]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:15 smithi153 ceph-mon[67960]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:15 smithi153 ceph-mon[67960]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:15 smithi153 ceph-mon[67960]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T04:00:15.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:15 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1817872109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:15.495 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[88407]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.495 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[88407]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.495 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[88407]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.495 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[88407]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T04:00:15.495 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1817872109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:15.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[86021]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[86021]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[86021]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:15.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[86021]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-12T04:00:15.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:15 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1817872109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:16 smithi153 ceph-mon[67960]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:16 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:16.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:16 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:16 smithi098 ceph-mon[86021]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:16 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:16.454 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:16 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:16.454 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:16 smithi098 ceph-mon[88407]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:16.454 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:16 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:16.454 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:16 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:16.823 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:16 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:00:16.451+0000 7f79ceb0e700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-12T04:00:17.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:17 smithi098 ceph-mon[86021]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:17.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:17 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:17.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:17 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:17.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:17 smithi098 ceph-mon[88407]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:17.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:17 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:17.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:17 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:17 smithi153 ceph-mon[67960]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:17 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:17.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:17 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:18.513 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:18 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:18.513 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:18 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:18 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:19.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:19 smithi098 ceph-mon[86021]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:19 smithi098 ceph-mon[88407]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:19 smithi153 ceph-mon[67960]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:20.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:00:20] "GET /metrics HTTP/1.1" 200 38013 "" "Prometheus/2.43.0" 2024-03-12T04:00:21.383 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:21.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:21.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:21.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:21.385 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:21.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:22.546 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: Detected new or changed devices on smithi098 2024-03-12T04:00:22.546 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: Reconfiguring mon.a (monmap changed)... 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: Reconfiguring daemon mon.a on smithi098 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: Detected new or changed devices on smithi098 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: Reconfiguring mon.a (monmap changed)... 2024-03-12T04:00:22.547 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: Reconfiguring daemon mon.a on smithi098 2024-03-12T04:00:22.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:22.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:22.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T04:00:22.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T04:00:22.548 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:22.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: Detected new or changed devices on smithi098 2024-03-12T04:00:22.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: Reconfiguring mon.a (monmap changed)... 2024-03-12T04:00:22.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: Reconfiguring daemon mon.a on smithi098 2024-03-12T04:00:22.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:22.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:22.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T04:00:22.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T04:00:22.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:23 smithi153 ceph-mon[67960]: Reconfiguring mgr.y (monmap changed)... 2024-03-12T04:00:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:23 smithi153 ceph-mon[67960]: Reconfiguring daemon mgr.y on smithi098 2024-03-12T04:00:23.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:23 smithi153 ceph-mon[67960]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:23 smithi098 ceph-mon[86021]: Reconfiguring mgr.y (monmap changed)... 2024-03-12T04:00:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:23 smithi098 ceph-mon[86021]: Reconfiguring daemon mgr.y on smithi098 2024-03-12T04:00:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:23 smithi098 ceph-mon[86021]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:23 smithi098 ceph-mon[88407]: Reconfiguring mgr.y (monmap changed)... 2024-03-12T04:00:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:23 smithi098 ceph-mon[88407]: Reconfiguring daemon mgr.y on smithi098 2024-03-12T04:00:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:23 smithi098 ceph-mon[88407]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:24.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:24.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:24.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: Reconfiguring mon.c (monmap changed)... 2024-03-12T04:00:24.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:24.760 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:24.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:24.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: Reconfiguring daemon mon.c on smithi098 2024-03-12T04:00:24.761 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[86021]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:24.761 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:24.761 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:24.762 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: Reconfiguring mon.c (monmap changed)... 2024-03-12T04:00:24.762 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:24.762 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:24.762 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:24.762 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: Reconfiguring daemon mon.c on smithi098 2024-03-12T04:00:24.762 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:24 smithi098 ceph-mon[88407]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: Reconfiguring mon.c (monmap changed)... 2024-03-12T04:00:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: Reconfiguring daemon mon.c on smithi098 2024-03-12T04:00:24.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:24 smithi153 ceph-mon[67960]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:26.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[88407]: Reconfiguring osd.0 (monmap changed)... 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.0 on smithi098 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[86021]: Reconfiguring osd.0 (monmap changed)... 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:26.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:26 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.0 on smithi098 2024-03-12T04:00:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:26 smithi153 ceph-mon[67960]: Reconfiguring osd.0 (monmap changed)... 2024-03-12T04:00:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T04:00:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:26 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.0 on smithi098 2024-03-12T04:00:27.415 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[88407]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:27.415 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[86021]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T04:00:27.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:27 smithi153 ceph-mon[67960]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T04:00:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:28.183 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:28 smithi098 ceph-mon[86021]: Reconfiguring osd.1 (monmap changed)... 2024-03-12T04:00:28.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:28 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.1 on smithi098 2024-03-12T04:00:28.184 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:28 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:28.184 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:28 smithi098 ceph-mon[88407]: Reconfiguring osd.1 (monmap changed)... 2024-03-12T04:00:28.184 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:28 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.1 on smithi098 2024-03-12T04:00:28.184 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:28 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:28 smithi153 ceph-mon[67960]: Reconfiguring osd.1 (monmap changed)... 2024-03-12T04:00:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:28 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.1 on smithi098 2024-03-12T04:00:28.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:28 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:29.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[86021]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[88407]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:29.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T04:00:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:29 smithi153 ceph-mon[67960]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T04:00:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:30.500 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:00:30] "GET /metrics HTTP/1.1" 200 38024 "" "Prometheus/2.43.0" 2024-03-12T04:00:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[86021]: Reconfiguring osd.2 (monmap changed)... 2024-03-12T04:00:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.2 on smithi098 2024-03-12T04:00:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T04:00:30.501 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:30.501 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[88407]: Reconfiguring osd.2 (monmap changed)... 2024-03-12T04:00:30.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.2 on smithi098 2024-03-12T04:00:30.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:30.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:30.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T04:00:30.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:30 smithi153 ceph-mon[67960]: Reconfiguring osd.2 (monmap changed)... 2024-03-12T04:00:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:30 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.2 on smithi098 2024-03-12T04:00:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T04:00:30.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:31.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:31 smithi098 ceph-mon[86021]: Reconfiguring osd.3 (monmap changed)... 2024-03-12T04:00:31.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:31 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.3 on smithi098 2024-03-12T04:00:31.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:31 smithi098 ceph-mon[86021]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:31.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:31 smithi098 ceph-mon[88407]: Reconfiguring osd.3 (monmap changed)... 2024-03-12T04:00:31.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:31 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.3 on smithi098 2024-03-12T04:00:31.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:31 smithi098 ceph-mon[88407]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:31 smithi153 ceph-mon[67960]: Reconfiguring osd.3 (monmap changed)... 2024-03-12T04:00:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:31 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.3 on smithi098 2024-03-12T04:00:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:31 smithi153 ceph-mon[67960]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[86021]: Reconfiguring rgw.foo.smithi098.tgzbnw (monmap changed)... 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[86021]: Reconfiguring daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[86021]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:32.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:32.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:32.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[88407]: Reconfiguring rgw.foo.smithi098.tgzbnw (monmap changed)... 2024-03-12T04:00:32.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:00:32.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:32.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[88407]: Reconfiguring daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T04:00:32.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:32 smithi098 ceph-mon[88407]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:32 smithi153 ceph-mon[67960]: Reconfiguring rgw.foo.smithi098.tgzbnw (monmap changed)... 2024-03-12T04:00:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:00:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:32 smithi153 ceph-mon[67960]: Reconfiguring daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T04:00:32.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:32 smithi153 ceph-mon[67960]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:34.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[86021]: Reconfiguring mon.b (monmap changed)... 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[86021]: Reconfiguring daemon mon.b on smithi153 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[88407]: Reconfiguring mon.b (monmap changed)... 2024-03-12T04:00:34.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:34.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:34.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:34.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:33 smithi098 ceph-mon[88407]: Reconfiguring daemon mon.b on smithi153 2024-03-12T04:00:34.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:34.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:34.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:33 smithi153 ceph-mon[67960]: Reconfiguring mon.b (monmap changed)... 2024-03-12T04:00:34.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-12T04:00:34.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-12T04:00:34.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:34.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:33 smithi153 ceph-mon[67960]: Reconfiguring daemon mon.b on smithi153 2024-03-12T04:00:35.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[86021]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[88407]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:35.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:35.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T04:00:35.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T04:00:35.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:35.079 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:34 smithi153 ceph-mon[67960]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:35.079 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:35.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:35.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-12T04:00:35.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-12T04:00:35.080 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:36.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:35 smithi098 ceph-mon[86021]: Reconfiguring mgr.x (monmap changed)... 2024-03-12T04:00:36.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:35 smithi098 ceph-mon[86021]: Reconfiguring daemon mgr.x on smithi153 2024-03-12T04:00:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:35 smithi098 ceph-mon[88407]: Reconfiguring mgr.x (monmap changed)... 2024-03-12T04:00:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:35 smithi098 ceph-mon[88407]: Reconfiguring daemon mgr.x on smithi153 2024-03-12T04:00:36.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:35 smithi153 ceph-mon[67960]: Reconfiguring mgr.x (monmap changed)... 2024-03-12T04:00:36.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:35 smithi153 ceph-mon[67960]: Reconfiguring daemon mgr.x on smithi153 2024-03-12T04:00:37.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:36 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:37.081 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:36 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:37.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:36 smithi153 ceph-mon[67960]: Reconfiguring osd.4 (monmap changed)... 2024-03-12T04:00:37.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:36 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T04:00:37.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:36 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:37.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:36 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.4 on smithi153 2024-03-12T04:00:37.082 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:36 smithi153 ceph-mon[67960]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[86021]: Reconfiguring osd.4 (monmap changed)... 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.4 on smithi153 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[86021]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:37.163 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:37.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:37.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[88407]: Reconfiguring osd.4 (monmap changed)... 2024-03-12T04:00:37.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T04:00:37.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:37.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.4 on smithi153 2024-03-12T04:00:37.164 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:36 smithi098 ceph-mon[88407]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:38.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:37 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:38.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:38.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:38.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T04:00:38.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:38.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T04:00:38.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:39.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:38 smithi153 ceph-mon[67960]: Reconfiguring osd.5 (monmap changed)... 2024-03-12T04:00:39.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:38 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.5 on smithi153 2024-03-12T04:00:39.005 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:38 smithi153 ceph-mon[67960]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:39.226 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:38 smithi098 ceph-mon[86021]: Reconfiguring osd.5 (monmap changed)... 2024-03-12T04:00:39.226 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:38 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.5 on smithi153 2024-03-12T04:00:39.226 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:38 smithi098 ceph-mon[86021]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:39.226 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:38 smithi098 ceph-mon[88407]: Reconfiguring osd.5 (monmap changed)... 2024-03-12T04:00:39.226 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:38 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.5 on smithi153 2024-03-12T04:00:39.226 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:38 smithi098 ceph-mon[88407]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:40 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:40 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:40 smithi153 ceph-mon[67960]: Reconfiguring osd.6 (monmap changed)... 2024-03-12T04:00:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:40 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T04:00:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:40 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:40 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.6 on smithi153 2024-03-12T04:00:40.474 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:00:40] "GET /metrics HTTP/1.1" 200 38024 "" "Prometheus/2.43.0" 2024-03-12T04:00:40.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:40.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:40.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[86021]: Reconfiguring osd.6 (monmap changed)... 2024-03-12T04:00:40.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T04:00:40.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:40.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.6 on smithi153 2024-03-12T04:00:40.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:40.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:40.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[88407]: Reconfiguring osd.6 (monmap changed)... 2024-03-12T04:00:40.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T04:00:40.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:40.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:40 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.6 on smithi153 2024-03-12T04:00:41.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:41 smithi153 ceph-mon[67960]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:41.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:41.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:41.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T04:00:41.391 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:41.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[86021]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:41.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[88407]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T04:00:41.403 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:42.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:42 smithi153 ceph-mon[67960]: Reconfiguring osd.7 (monmap changed)... 2024-03-12T04:00:42.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:42 smithi153 ceph-mon[67960]: Reconfiguring daemon osd.7 on smithi153 2024-03-12T04:00:42.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:42 smithi098 ceph-mon[86021]: Reconfiguring osd.7 (monmap changed)... 2024-03-12T04:00:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:42 smithi098 ceph-mon[86021]: Reconfiguring daemon osd.7 on smithi153 2024-03-12T04:00:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:42 smithi098 ceph-mon[88407]: Reconfiguring osd.7 (monmap changed)... 2024-03-12T04:00:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:42 smithi098 ceph-mon[88407]: Reconfiguring daemon osd.7 on smithi153 2024-03-12T04:00:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:43.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:43 smithi153 ceph-mon[67960]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:43.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:43 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:43.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:43 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:43.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:43 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:00:43.288 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:43 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[86021]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[88407]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:00:43.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:43 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: Reconfiguring rgw.foo.smithi153.tuboyg (monmap changed)... 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: Reconfiguring daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:00:44.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:44 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:00:44.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: Reconfiguring rgw.foo.smithi153.tuboyg (monmap changed)... 2024-03-12T04:00:44.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: Reconfiguring daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.576 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: Reconfiguring rgw.foo.smithi153.tuboyg (monmap changed)... 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: Reconfiguring daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-12T04:00:44.577 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.578 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.579 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.580 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-12T04:00:44.581 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:00:44.582 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:44 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mon 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all crash 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mds 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nfs 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all node-exporter 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all prometheus 2024-03-12T04:00:45.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all alertmanager 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all grafana 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all loki 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all promtail 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Finalizing container_image settings 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:00:45.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: Upgrade: Complete! 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:45.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:45 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:45.536 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mon 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all crash 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mds 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nfs 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all node-exporter 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all prometheus 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all alertmanager 2024-03-12T04:00:45.537 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all grafana 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all loki 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all promtail 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Finalizing container_image settings 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:00:45.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mon 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all crash 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mds 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:00:45.539 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nfs 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all node-exporter 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all prometheus 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all alertmanager 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all grafana 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all loki 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all promtail 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Finalizing container_image settings 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:00:45.540 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:00:45.541 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: Upgrade: Complete! 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:45.542 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: Upgrade: Complete! 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:00:45.544 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:45 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:45.841 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T04:00:46.396 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:46 smithi098 ceph-mon[86021]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:46.396 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:46 smithi098 ceph-mon[88407]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:46.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:46 smithi153 ceph-mon[67960]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:47.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:47 smithi098 ceph-mon[86021]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:47.428 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:47.429 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:47 smithi098 ceph-mon[88407]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:47.429 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:47 smithi153 ceph-mon[67960]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:00:47.752 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:00:47.752 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (3m) 31s ago 24m 28.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:00:47.752 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (2m) 47s ago 24m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:00:47.752 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (2m) 31s ago 24m 53.6M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (4m) 47s ago 29m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (13m) 31s ago 31m 495M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (61s) 31s ago 31m 38.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (110s) 47s ago 29m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (36s) 31s ago 29m 20.4M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (3m) 31s ago 25m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (3m) 47s ago 25m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (29m) 31s ago 29m 56.8M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (28m) 31s ago 28m 58.9M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (28m) 31s ago 28m 50.8M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (27m) 31s ago 27m 53.5M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (27m) 47s ago 27m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (27m) 47s ago 27m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (26m) 47s ago 26m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:00:47.753 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (26m) 47s ago 26m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:00:47.754 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (3m) 47s ago 25m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:00:47.754 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (24m) 31s ago 24m 77.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:00:47.754 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (24m) 47s ago 24m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:00:48.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:48 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:48.483 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:48 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:48.508 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-03-12T04:00:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:48 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:49.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:49 smithi098 ceph-mon[88407]: from='client.54174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:49 smithi098 ceph-mon[88407]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:49 smithi098 ceph-mon[86021]: from='client.54174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:49 smithi098 ceph-mon[86021]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:49 smithi153 ceph-mon[67960]: from='client.54174 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:49 smithi153 ceph-mon[67960]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:50.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:00:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:00:50] "GET /metrics HTTP/1.1" 200 38022 "" "Prometheus/2.43.0" 2024-03-12T04:00:50.615 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:00:51.248 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-03-12T04:00:51.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:51 smithi098 ceph-mon[86021]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:51.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:51 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1504909415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:51.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:51 smithi098 ceph-mon[88407]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:51.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:51 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1504909415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:51 smithi153 ceph-mon[67960]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:51 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1504909415' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:53.326 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)" 2024-03-12T04:00:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:53 smithi098 ceph-mon[86021]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:53 smithi098 ceph-mon[88407]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:53 smithi153 ceph-mon[67960]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:53.995 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-03-12T04:00:54.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:54 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2047171583' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:54.533 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:54 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2047171583' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:54 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2047171583' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:00:55.419 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:55 smithi098 ceph-mon[86021]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:55.419 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:55 smithi098 ceph-mon[88407]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:55 smithi153 ceph-mon[67960]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:56.496 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:56.497 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:56.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:00:57.428 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:00:57.429 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:57 smithi098 ceph-mon[86021]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:57.429 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:57 smithi098 ceph-mon[86021]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:57.429 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:57 smithi098 ceph-mon[88407]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:57.430 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:57 smithi098 ceph-mon[88407]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:57 smithi153 ceph-mon[67960]: from='client.54192 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:00:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:57 smithi153 ceph-mon[67960]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:00:57.997 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T04:00:58.307 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:58 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:58.308 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:58 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:58 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:00:59.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:00:59 smithi098 ceph-mon[86021]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:59.505 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:00:59 smithi098 ceph-mon[88407]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:00:59 smithi153 ceph-mon[67960]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:00:59.936 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:00:59.936 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": null, 2024-03-12T04:00:59.936 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": false, 2024-03-12T04:00:59.936 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "", 2024-03-12T04:00:59.936 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:00:59.936 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": null, 2024-03-12T04:00:59.936 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T04:00:59.937 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:00:59.937 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:01:00.519 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:01:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:01:00] "GET /metrics HTTP/1.1" 200 38023 "" "Prometheus/2.43.0" 2024-03-12T04:01:00.543 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T04:01:01.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:01 smithi098 ceph-mon[88407]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:01 smithi098 ceph-mon[88407]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:01 smithi098 ceph-mon[86021]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:01 smithi098 ceph-mon[86021]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:01 smithi153 ceph-mon[67960]: from='client.64132 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:01 smithi153 ceph-mon[67960]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:02.670 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T04:01:03.302 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-03-12T04:01:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:03 smithi098 ceph-mon[86021]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:03 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/536958130' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:01:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:03 smithi098 ceph-mon[88407]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:03 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/536958130' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:01:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:03 smithi153 ceph-mon[67960]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:03 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/536958130' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:01:05.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:05 smithi098 ceph-mon[86021]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:05.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:05 smithi098 ceph-mon[88407]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:05 smithi153 ceph-mon[67960]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:07.050 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:07.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:07 smithi098 ceph-mon[88407]: from='client.54210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:07.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:07 smithi098 ceph-mon[88407]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:07.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:07 smithi098 ceph-mon[86021]: from='client.54210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:07 smithi098 ceph-mon[86021]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:07.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:07 smithi153 ceph-mon[67960]: from='client.54210 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:07 smithi153 ceph-mon[67960]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:07.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T04:01:08.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[88407]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[86021]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:01:08.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:08 smithi153 ceph-mon[67960]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:08 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:01:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:09.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:09 smithi098 ceph-mon[86021]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:09.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:09 smithi098 ceph-mon[86021]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:09.493 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:09 smithi098 ceph-mon[88407]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:09.493 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:09 smithi098 ceph-mon[88407]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:09 smithi153 ceph-mon[67960]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:09 smithi153 ceph-mon[67960]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:09.772 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (3m) 54s ago 25m 28.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (3m) 69s ago 24m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (2m) 54s ago 24m 53.6M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (4m) 69s ago 29m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (14m) 54s ago 31m 495M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (84s) 54s ago 31m 38.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:01:10.433 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (2m) 69s ago 30m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (59s) 54s ago 30m 20.4M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (4m) 54s ago 26m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (4m) 69s ago 26m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (29m) 54s ago 29m 56.8M 2992M 17.2.0 e1d6a67b021e e54e5d5eddac 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (29m) 54s ago 29m 58.9M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (28m) 54s ago 28m 50.8M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (28m) 54s ago 28m 53.5M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (27m) 69s ago 27m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (27m) 69s ago 27m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (27m) 69s ago 27m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (26m) 69s ago 26m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:01:10.434 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (3m) 69s ago 26m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:01:10.435 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (24m) 54s ago 24m 77.5M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:01:10.435 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (24m) 69s ago 24m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:01:10.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:01:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:01:10] "GET /metrics HTTP/1.1" 200 38023 "" "Prometheus/2.43.0" 2024-03-12T04:01:10.847 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:01:10.847 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:01:10.847 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:01:10.847 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 5 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:01:10.848 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "0/8 daemons upgraded", 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:01:11.249 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:01:11.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='client.64150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:11.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:11.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:11.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2481207176' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:11.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='client.64150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2481207176' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='client.64150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2481207176' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:11.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:12.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:12.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mgr 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mon 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all crash 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: Upgrade: osd.0 is safe to restart 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[88407]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mgr 2024-03-12T04:01:12.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mon 2024-03-12T04:01:12.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all crash 2024-03-12T04:01:12.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-12T04:01:12.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: Upgrade: osd.0 is safe to restart 2024-03-12T04:01:12.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:12 smithi098 ceph-mon[86021]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:12.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:12.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:01:12.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:01:12.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mgr 2024-03-12T04:01:12.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mon 2024-03-12T04:01:12.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all crash 2024-03-12T04:01:12.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-12T04:01:12.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: Upgrade: osd.0 is safe to restart 2024-03-12T04:01:12.594 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:12 smithi153 ceph-mon[67960]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[86021]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T04:01:13.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[88407]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T04:01:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:13 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:13 smithi153 ceph-mon[67960]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:13 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:13 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-12T04:01:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:13 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:14.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:14 smithi098 ceph-mon[86021]: Upgrade: Updating osd.0 2024-03-12T04:01:14.402 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:14 smithi098 ceph-mon[86021]: Deploying daemon osd.0 on smithi098 2024-03-12T04:01:14.403 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:14 smithi098 ceph-mon[88407]: Upgrade: Updating osd.0 2024-03-12T04:01:14.403 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:14 smithi098 ceph-mon[88407]: Deploying daemon osd.0 on smithi098 2024-03-12T04:01:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:14 smithi153 ceph-mon[67960]: Upgrade: Updating osd.0 2024-03-12T04:01:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:14 smithi153 ceph-mon[67960]: Deploying daemon osd.0 on smithi098 2024-03-12T04:01:15.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:15 smithi098 ceph-mon[86021]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:15.467 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:15 smithi098 ceph-mon[88407]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:15 smithi153 ceph-mon[67960]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:16.283 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:15 smithi098 systemd[1]: Stopping Ceph osd.0 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:01:16.283 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:16 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[32344]: 2024-03-12T04:01:16.085+0000 7f68a3cc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:01:16.283 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:16 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[32344]: 2024-03-12T04:01:16.085+0000 7f68a3cc8700 -1 osd.0 113 *** Got signal Terminated *** 2024-03-12T04:01:16.284 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:16 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[32344]: 2024-03-12T04:01:16.085+0000 7f68a3cc8700 -1 osd.0 113 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:01:16.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:16 smithi098 ceph-mon[86021]: osd.0 marked itself down and dead 2024-03-12T04:01:16.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:16 smithi098 ceph-mon[88407]: osd.0 marked itself down and dead 2024-03-12T04:01:16.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:16 smithi153 ceph-mon[67960]: osd.0 marked itself down and dead 2024-03-12T04:01:16.979 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:16 smithi098 podman[95093]: 2024-03-12 04:01:16.709187359 +0000 UTC m=+0.722460805 container died e54e5d5eddac598e65f4f1e994ef626442869aaef7237b82e221dcd4e62ed663 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, vcs-type=git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, distribution-scope=public, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc.) 2024-03-12T04:01:17.284 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:17 smithi098 podman[95093]: 2024-03-12 04:01:17.18596121 +0000 UTC m=+1.199234698 container cleanup e54e5d5eddac598e65f4f1e994ef626442869aaef7237b82e221dcd4e62ed663 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-03-12T04:01:17.285 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:17 smithi098 bash[95093]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0 2024-03-12T04:01:17.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:17 smithi098 ceph-mon[88407]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:17.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:17 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:01:17.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:17 smithi098 ceph-mon[88407]: osdmap e114: 8 total, 7 up, 8 in 2024-03-12T04:01:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:17 smithi098 ceph-mon[86021]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:17 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:01:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:17 smithi098 ceph-mon[86021]: osdmap e114: 8 total, 7 up, 8 in 2024-03-12T04:01:17.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:17 smithi098 podman[95110]: 2024-03-12 04:01:17.42917201 +0000 UTC m=+0.718121296 container remove e54e5d5eddac598e65f4f1e994ef626442869aaef7237b82e221dcd4e62ed663 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.expose-services=, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, RELEASE=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, ceph=True, distribution-scope=public, GIT_BRANCH=HEAD, architecture=x86_64) 2024-03-12T04:01:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:17 smithi153 ceph-mon[67960]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:17 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:01:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:17 smithi153 ceph-mon[67960]: osdmap e114: 8 total, 7 up, 8 in 2024-03-12T04:01:18.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:17 smithi098 podman[95198]: 2024-03-12 04:01:17.755944495 +0000 UTC m=+0.106810244 container create 6c982060899439c8ff92226b7e83dc72f5f1785d7dbb2f6417ca866c3303c69b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-12T04:01:18.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:17 smithi098 podman[95198]: 2024-03-12 04:01:17.668151541 +0000 UTC m=+0.019017291 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:18.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:17 smithi098 podman[95198]: 2024-03-12 04:01:17.974676734 +0000 UTC m=+0.325542497 container init 6c982060899439c8ff92226b7e83dc72f5f1785d7dbb2f6417ca866c3303c69b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T04:01:18.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:17 smithi098 podman[95198]: 2024-03-12 04:01:17.977828004 +0000 UTC m=+0.328693763 container start 6c982060899439c8ff92226b7e83dc72f5f1785d7dbb2f6417ca866c3303c69b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-12T04:01:18.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:18 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:18.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:18 smithi098 ceph-mon[88407]: osdmap e115: 8 total, 7 up, 8 in 2024-03-12T04:01:18.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:18 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:18.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:18 smithi098 ceph-mon[86021]: osdmap e115: 8 total, 7 up, 8 in 2024-03-12T04:01:18.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:18 smithi098 podman[95198]: 2024-03-12 04:01:18.074590219 +0000 UTC m=+0.425455985 container attach 6c982060899439c8ff92226b7e83dc72f5f1785d7dbb2f6417ca866c3303c69b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, ceph=True) 2024-03-12T04:01:18.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:18 smithi098 podman[95198]: 2024-03-12 04:01:18.266625875 +0000 UTC m=+0.617491635 container died 6c982060899439c8ff92226b7e83dc72f5f1785d7dbb2f6417ca866c3303c69b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e) 2024-03-12T04:01:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:18 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:18 smithi153 ceph-mon[67960]: osdmap e115: 8 total, 7 up, 8 in 2024-03-12T04:01:19.073 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:18 smithi098 podman[95224]: 2024-03-12 04:01:18.954113575 +0000 UTC m=+0.679022583 container remove 6c982060899439c8ff92226b7e83dc72f5f1785d7dbb2f6417ca866c3303c69b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-12T04:01:19.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:18 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-12T04:01:19.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:18 smithi098 systemd[1]: Stopped Ceph osd.0 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:01:19.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:18 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.0.service: Consumed 13.841s CPU time. 2024-03-12T04:01:19.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:19 smithi098 ceph-mon[86021]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:01:19.515 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:19 smithi098 ceph-mon[86021]: Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-03-12T04:01:19.516 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:19 smithi098 ceph-mon[88407]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:01:19.516 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:19 smithi098 ceph-mon[88407]: Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-03-12T04:01:19.516 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:19 smithi098 systemd[1]: Starting Ceph osd.0 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:01:19.516 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:19 smithi098 podman[95330]: 2024-03-12 04:01:19.312204701 +0000 UTC m=+0.017995762 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:19 smithi153 ceph-mon[67960]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:01:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:19 smithi153 ceph-mon[67960]: Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-03-12T04:01:19.823 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:19 smithi098 podman[95330]: 2024-03-12 04:01:19.513864073 +0000 UTC m=+0.219655193 container create 80d4fd7b587a9b62bec66cb045fb21297f552240313f909ddc25d4bd0d30d844 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:01:19.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:19 smithi098 podman[95330]: 2024-03-12 04:01:19.742155219 +0000 UTC m=+0.447946276 container init 80d4fd7b587a9b62bec66cb045fb21297f552240313f909ddc25d4bd0d30d844 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-12T04:01:19.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:19 smithi098 podman[95330]: 2024-03-12 04:01:19.745401072 +0000 UTC m=+0.451192126 container start 80d4fd7b587a9b62bec66cb045fb21297f552240313f909ddc25d4bd0d30d844 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-12T04:01:20.186 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:19 smithi098 podman[95330]: 2024-03-12 04:01:19.845522671 +0000 UTC m=+0.551313723 container attach 80d4fd7b587a9b62bec66cb045fb21297f552240313f909ddc25d4bd0d30d844 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_BRANCH=HEAD) 2024-03-12T04:01:20.513 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:01:20] "GET /metrics HTTP/1.1" 200 38020 "" "Prometheus/2.43.0" 2024-03-12T04:01:20.513 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate[95345]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T04:01:20.513 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 bash[95330]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T04:01:20.513 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 bash[95330]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 bash[95330]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 bash[95330]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 bash[95330]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 bash[95330]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 bash[95330]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate[95345]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate[95345]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate[95345]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate[95345]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate[95345]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate[95345]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-12T04:01:20.514 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 podman[95471]: 2024-03-12 04:01:20.236140989 +0000 UTC m=+0.018271743 container died 80d4fd7b587a9b62bec66cb045fb21297f552240313f909ddc25d4bd0d30d844 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T04:01:21.073 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:20 smithi098 podman[95471]: 2024-03-12 04:01:20.817494784 +0000 UTC m=+0.599625590 container remove 80d4fd7b587a9b62bec66cb045fb21297f552240313f909ddc25d4bd0d30d844 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-03-12T04:01:21.073 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 podman[95538]: 2024-03-12 04:01:20.932423188 +0000 UTC m=+0.018285824 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:21.074 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 podman[95538]: 2024-03-12 04:01:21.046580404 +0000 UTC m=+0.132443081 container create a109e6088552df25af19a200f56aa63309f8047084104f49df3587286e30ec00 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-12T04:01:21.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:21 smithi098 ceph-mon[86021]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:21.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:21 smithi098 ceph-mon[88407]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:21.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 podman[95538]: 2024-03-12 04:01:21.290962599 +0000 UTC m=+0.376825230 container init a109e6088552df25af19a200f56aa63309f8047084104f49df3587286e30ec00 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-12T04:01:21.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 podman[95538]: 2024-03-12 04:01:21.293998636 +0000 UTC m=+0.379861268 container start a109e6088552df25af19a200f56aa63309f8047084104f49df3587286e30ec00 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-12T04:01:21.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 ceph-osd[95558]: -- 172.21.15.98:0/2459497612 <== mon.1 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557a3749f680 con 0x557a37469000 2024-03-12T04:01:21.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 bash[95538]: a109e6088552df25af19a200f56aa63309f8047084104f49df3587286e30ec00 2024-03-12T04:01:21.574 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 systemd[1]: Started Ceph osd.0 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:01:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:21 smithi153 ceph-mon[67960]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:22.323 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:21 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[95554]: 2024-03-12T04:01:21.847+0000 7f460260e7c0 -1 Falling back to public interface 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[86021]: pgmap v48: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.798 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.799 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:22 smithi098 ceph-mon[88407]: pgmap v48: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:22.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:22.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:22 smithi153 ceph-mon[67960]: pgmap v48: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:23.733 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:23 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 78/723 objects degraded (10.788%), 24 pgs degraded (PG_DEGRADED) 2024-03-12T04:01:23.733 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:23 smithi098 ceph-mon[86021]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2024-03-12T04:01:23.734 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:23 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 78/723 objects degraded (10.788%), 24 pgs degraded (PG_DEGRADED) 2024-03-12T04:01:23.734 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:23 smithi098 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2024-03-12T04:01:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:23 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 78/723 objects degraded (10.788%), 24 pgs degraded (PG_DEGRADED) 2024-03-12T04:01:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:23 smithi153 ceph-mon[67960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2024-03-12T04:01:24.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:24 smithi098 ceph-mon[86021]: pgmap v49: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:24.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:24 smithi098 ceph-mon[88407]: pgmap v49: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:24.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[95554]: 2024-03-12T04:01:24.575+0000 7f460260e7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-12T04:01:24.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[95554]: 2024-03-12T04:01:24.603+0000 7f460260e7c0 -1 osd.0 113 log_to_monitors true 2024-03-12T04:01:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:24 smithi153 ceph-mon[67960]: pgmap v49: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:25.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:25 smithi098 ceph-mon[86021]: from='osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T04:01:25.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:25 smithi098 ceph-mon[86021]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T04:01:25.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:25 smithi098 ceph-mon[88407]: from='osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T04:01:25.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:25 smithi098 ceph-mon[88407]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T04:01:25.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:01:25 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[95554]: 2024-03-12T04:01:25.484+0000 7f45f3a66700 -1 osd.0 113 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:01:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:25 smithi153 ceph-mon[67960]: from='osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T04:01:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:25 smithi153 ceph-mon[67960]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: osdmap e116: 8 total, 7 up, 8 in 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: from='osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: pgmap v51: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: osdmap e116: 8 total, 7 up, 8 in 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: from='osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: pgmap v51: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-12T04:01:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: osdmap e116: 8 total, 7 up, 8 in 2024-03-12T04:01:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: from='osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:01:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:01:26.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: pgmap v51: 161 pgs: 24 active+undersized+degraded, 34 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s; 78/723 objects degraded (10.788%) 2024-03-12T04:01:26.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:26.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:26.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:27.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:01:27.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[88407]: osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045] boot 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[88407]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[86021]: osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045] boot 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[86021]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:27.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:27 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:01:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:27 smithi153 ceph-mon[67960]: osd.0 [v2:172.21.15.98:6802/41968045,v1:172.21.15.98:6803/41968045] boot 2024-03-12T04:01:27.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:27 smithi153 ceph-mon[67960]: osdmap e117: 8 total, 8 up, 8 in 2024-03-12T04:01:27.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-12T04:01:27.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:27 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:27.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:27.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:28.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:28 smithi098 ceph-mon[86021]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T04:01:28.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:28 smithi098 ceph-mon[86021]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-03-12T04:01:28.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:28 smithi098 ceph-mon[88407]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T04:01:28.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:28 smithi098 ceph-mon[88407]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-03-12T04:01:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:28 smithi153 ceph-mon[67960]: osdmap e118: 8 total, 8 up, 8 in 2024-03-12T04:01:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:28 smithi153 ceph-mon[67960]: pgmap v54: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-03-12T04:01:29.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:29 smithi098 ceph-mon[86021]: Health check update: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T04:01:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:29 smithi098 ceph-mon[88407]: Health check update: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T04:01:29.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:29 smithi153 ceph-mon[67960]: Health check update: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded (PG_DEGRADED) 2024-03-12T04:01:30.485 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:01:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:01:30] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.43.0" 2024-03-12T04:01:30.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:30 smithi098 ceph-mon[86021]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-03-12T04:01:30.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:30 smithi098 ceph-mon[88407]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-03-12T04:01:30.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:30 smithi153 ceph-mon[67960]: pgmap v55: 161 pgs: 37 peering, 8 active+undersized+degraded, 13 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-03-12T04:01:31.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:01:31 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:01:31.172+0000 7f79ceb0e700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[86021]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:32.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:32.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:32 smithi098 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:32.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:32.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:32.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:32 smithi153 ceph-mon[67960]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:01:33.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:33 smithi098 ceph-mon[86021]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T04:01:33.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:33 smithi098 ceph-mon[88407]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T04:01:33.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:33 smithi153 ceph-mon[67960]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-12T04:01:34.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:34 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded) 2024-03-12T04:01:34.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:34 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:01:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:34 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded) 2024-03-12T04:01:34.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:34 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:01:34.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:34 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded) 2024-03-12T04:01:34.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:34 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:01:35.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:35 smithi098 ceph-mon[86021]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:35.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:35 smithi098 ceph-mon[88407]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:35.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:35 smithi153 ceph-mon[67960]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:37.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:37 smithi098 ceph-mon[86021]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T04:01:37.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:37 smithi098 ceph-mon[88407]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T04:01:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:37 smithi153 ceph-mon[67960]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T04:01:38.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:38 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:38.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:38 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:38 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:39.474 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:39 smithi098 ceph-mon[86021]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-03-12T04:01:39.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:39 smithi098 ceph-mon[88407]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-03-12T04:01:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:39 smithi153 ceph-mon[67960]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-03-12T04:01:40.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:01:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:01:40] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.43.0" 2024-03-12T04:01:41.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:41 smithi098 ceph-mon[86021]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:41 smithi098 ceph-mon[88407]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:41.600 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:01:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:41 smithi153 ceph-mon[67960]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (4m) 15s ago 25m 28.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (3m) 101s ago 25m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (3m) 15s ago 24m 56.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (5m) 101s ago 30m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (14m) 15s ago 32m 502M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (116s) 15s ago 32m 41.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:01:42.245 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (2m) 101s ago 30m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (91s) 15s ago 30m 33.7M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (4m) 15s ago 26m 15.1M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (4m) 101s ago 26m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (21s) 15s ago 30m 12.3M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (29m) 15s ago 29m 59.7M 2992M 17.2.0 e1d6a67b021e 91ff13bbeff8 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (29m) 15s ago 29m 51.4M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (28m) 15s ago 28m 54.1M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (28m) 101s ago 28m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (28m) 101s ago 28m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:01:42.246 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (27m) 101s ago 27m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:01:42.247 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (27m) 101s ago 27m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:01:42.247 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (4m) 101s ago 26m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:01:42.247 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (25m) 15s ago 25m 77.8M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:01:42.247 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (25m) 101s ago 25m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:01:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:42.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:42.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:42.593 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:42.633 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:01:42.633 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 1 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:01:42.634 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:01:42.635 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-12T04:01:42.635 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 6 2024-03-12T04:01:42.635 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:01:42.635 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "1/8 daemons upgraded", 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:01:42.962 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:01:42.963 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:01:43.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[86021]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[86021]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[86021]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[86021]: from='client.64198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/3219412577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[88407]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[88407]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[88407]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[88407]: from='client.64198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:43 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/3219412577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:43 smithi153 ceph-mon[67960]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:43 smithi153 ceph-mon[67960]: from='client.54258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:43 smithi153 ceph-mon[67960]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:43 smithi153 ceph-mon[67960]: from='client.64198 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:43 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/3219412577' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:01:44.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:44 smithi098 ceph-mon[86021]: from='client.54276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:44 smithi098 ceph-mon[88407]: from='client.54276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:44.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:44 smithi153 ceph-mon[67960]: from='client.54276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:01:45.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:45 smithi098 ceph-mon[86021]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:45.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:45 smithi098 ceph-mon[88407]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:45 smithi153 ceph-mon[67960]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:46.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:46 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:46.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:46 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:46.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:46 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:47.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:47 smithi098 ceph-mon[86021]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:47.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:47 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:47.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:47 smithi098 ceph-mon[86021]: Upgrade: osd.1 is safe to restart 2024-03-12T04:01:47.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:47 smithi098 ceph-mon[88407]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:47.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:47 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:47.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:47 smithi098 ceph-mon[88407]: Upgrade: osd.1 is safe to restart 2024-03-12T04:01:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:47 smithi153 ceph-mon[67960]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:47 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-12T04:01:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:47 smithi153 ceph-mon[67960]: Upgrade: osd.1 is safe to restart 2024-03-12T04:01:48.569 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:48.570 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:48.570 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T04:01:48.570 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:48.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:48.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:48.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T04:01:48.570 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:48 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:48 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:48 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:48 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-12T04:01:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:48 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:01:49.572 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:49 smithi098 ceph-mon[86021]: Upgrade: Updating osd.1 2024-03-12T04:01:49.572 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:49 smithi098 ceph-mon[86021]: Deploying daemon osd.1 on smithi098 2024-03-12T04:01:49.572 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:49 smithi098 ceph-mon[86021]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:49.572 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:49 smithi098 ceph-mon[88407]: Upgrade: Updating osd.1 2024-03-12T04:01:49.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:49 smithi098 ceph-mon[88407]: Deploying daemon osd.1 on smithi098 2024-03-12T04:01:49.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:49 smithi098 ceph-mon[88407]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:49 smithi153 ceph-mon[67960]: Upgrade: Updating osd.1 2024-03-12T04:01:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:49 smithi153 ceph-mon[67960]: Deploying daemon osd.1 on smithi098 2024-03-12T04:01:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:49 smithi153 ceph-mon[67960]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:01:50.350 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:01:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:01:50] "GET /metrics HTTP/1.1" 200 38155 "" "Prometheus/2.43.0" 2024-03-12T04:01:51.073 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:50 smithi098 systemd[1]: Stopping Ceph osd.1 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:01:51.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:51 smithi098 ceph-mon[88407]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:51.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:51 smithi098 ceph-mon[88407]: osd.1 marked itself down and dead 2024-03-12T04:01:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:51 smithi098 ceph-mon[86021]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:51 smithi098 ceph-mon[86021]: osd.1 marked itself down and dead 2024-03-12T04:01:51.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[35306]: 2024-03-12T04:01:51.169+0000 7f34da9d7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:01:51.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[35306]: 2024-03-12T04:01:51.169+0000 7f34da9d7700 -1 osd.1 118 *** Got signal Terminated *** 2024-03-12T04:01:51.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[35306]: 2024-03-12T04:01:51.169+0000 7f34da9d7700 -1 osd.1 118 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:01:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:51 smithi153 ceph-mon[67960]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:01:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:51 smithi153 ceph-mon[67960]: osd.1 marked itself down and dead 2024-03-12T04:01:51.955 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:51 smithi098 podman[99944]: 2024-03-12 04:01:51.598648731 +0000 UTC m=+0.531254620 container died 91ff13bbeff8668b8d8cb498efafd660d1d4ea72f128f491693a65dcfcf2787a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, version=8, ceph=True, io.buildah.version=1.19.8, architecture=x86_64, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, name=centos-stream, release=754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T04:01:52.269 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:52 smithi098 podman[99944]: 2024-03-12 04:01:52.153232108 +0000 UTC m=+1.085838034 container cleanup 91ff13bbeff8668b8d8cb498efafd660d1d4ea72f128f491693a65dcfcf2787a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, name=centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vcs-type=git, RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870) 2024-03-12T04:01:52.269 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:52 smithi098 bash[99944]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1 2024-03-12T04:01:52.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:52 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:01:52.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:52 smithi098 ceph-mon[86021]: osdmap e119: 8 total, 7 up, 8 in 2024-03-12T04:01:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:52 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:01:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:52 smithi098 ceph-mon[88407]: osdmap e119: 8 total, 7 up, 8 in 2024-03-12T04:01:52.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:52 smithi098 podman[99961]: 2024-03-12 04:01:52.446476898 +0000 UTC m=+0.844750801 container remove 91ff13bbeff8668b8d8cb498efafd660d1d4ea72f128f491693a65dcfcf2787a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, name=centos-stream, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-03-12T04:01:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:52 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:01:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:52 smithi153 ceph-mon[67960]: osdmap e119: 8 total, 7 up, 8 in 2024-03-12T04:01:53.073 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:52 smithi098 podman[100056]: 2024-03-12 04:01:52.764143985 +0000 UTC m=+0.114608638 container create 6799bc02b5c0f62a3806ca83d7dbe82f13ce0a13111ad26c5004f7fdb11a9882 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-12T04:01:53.074 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:52 smithi098 podman[100056]: 2024-03-12 04:01:52.668090705 +0000 UTC m=+0.018555317 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:53.074 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:52 smithi098 podman[100056]: 2024-03-12 04:01:52.983532256 +0000 UTC m=+0.333996881 container init 6799bc02b5c0f62a3806ca83d7dbe82f13ce0a13111ad26c5004f7fdb11a9882 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:01:53.074 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:52 smithi098 podman[100056]: 2024-03-12 04:01:52.987175095 +0000 UTC m=+0.337639706 container start 6799bc02b5c0f62a3806ca83d7dbe82f13ce0a13111ad26c5004f7fdb11a9882 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T04:01:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:53 smithi098 ceph-mon[86021]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:53 smithi098 ceph-mon[86021]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-03-12T04:01:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:53 smithi098 ceph-mon[86021]: osdmap e120: 8 total, 7 up, 8 in 2024-03-12T04:01:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:53 smithi098 ceph-mon[88407]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:53 smithi098 ceph-mon[88407]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-03-12T04:01:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:53 smithi098 ceph-mon[88407]: osdmap e120: 8 total, 7 up, 8 in 2024-03-12T04:01:53.575 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:53 smithi098 podman[100056]: 2024-03-12 04:01:53.075182678 +0000 UTC m=+0.425647294 container attach 6799bc02b5c0f62a3806ca83d7dbe82f13ce0a13111ad26c5004f7fdb11a9882 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-12T04:01:53.575 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:53 smithi098 podman[100056]: 2024-03-12 04:01:53.288479403 +0000 UTC m=+0.638944009 container died 6799bc02b5c0f62a3806ca83d7dbe82f13ce0a13111ad26c5004f7fdb11a9882 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:01:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:53 smithi153 ceph-mon[67960]: pgmap v67: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:53 smithi153 ceph-mon[67960]: Health check failed: Reduced data availability: 9 pgs inactive, 15 pgs peering (PG_AVAILABILITY) 2024-03-12T04:01:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:53 smithi153 ceph-mon[67960]: osdmap e120: 8 total, 7 up, 8 in 2024-03-12T04:01:54.323 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 podman[100082]: 2024-03-12 04:01:54.037950847 +0000 UTC m=+0.743214940 container remove 6799bc02b5c0f62a3806ca83d7dbe82f13ce0a13111ad26c5004f7fdb11a9882 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:01:54.324 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-12T04:01:54.324 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 systemd[1]: Stopped Ceph osd.1 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:01:54.324 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.1.service: Consumed 14.415s CPU time. 2024-03-12T04:01:54.324 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 systemd[1]: Starting Ceph osd.1 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:01:54.810 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 podman[100182]: 2024-03-12 04:01:54.381919317 +0000 UTC m=+0.018099928 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:54.810 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 podman[100182]: 2024-03-12 04:01:54.497233117 +0000 UTC m=+0.133413762 container create 2edeef89e29b5e42f9db4a8397431cedd6d2eccdb60e42c3b772cb0521af9b8d (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:01:54.810 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 podman[100182]: 2024-03-12 04:01:54.716915445 +0000 UTC m=+0.353096060 container init 2edeef89e29b5e42f9db4a8397431cedd6d2eccdb60e42c3b772cb0521af9b8d (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-12T04:01:54.810 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 podman[100182]: 2024-03-12 04:01:54.72028238 +0000 UTC m=+0.356462992 container start 2edeef89e29b5e42f9db4a8397431cedd6d2eccdb60e42c3b772cb0521af9b8d (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-12T04:01:54.810 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:54 smithi098 podman[100182]: 2024-03-12 04:01:54.808367477 +0000 UTC m=+0.444548093 container attach 2edeef89e29b5e42f9db4a8397431cedd6d2eccdb60e42c3b772cb0521af9b8d (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:01:55.502 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-mon[86021]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:55.502 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-mon[88407]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:55.502 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate[100197]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate[100197]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 bash[100182]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 bash[100182]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 bash[100182]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 bash[100182]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 bash[100182]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 bash[100182]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 bash[100182]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate[100197]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate[100197]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate[100197]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate[100197]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate[100197]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-12T04:01:55.503 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 podman[100182]: 2024-03-12 04:01:55.182577706 +0000 UTC m=+0.818758315 container died 2edeef89e29b5e42f9db4a8397431cedd6d2eccdb60e42c3b772cb0521af9b8d (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T04:01:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:55 smithi153 ceph-mon[67960]: pgmap v69: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:01:56.073 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:55 smithi098 podman[100317]: 2024-03-12 04:01:55.795660432 +0000 UTC m=+0.608156644 container remove 2edeef89e29b5e42f9db4a8397431cedd6d2eccdb60e42c3b772cb0521af9b8d (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-12T04:01:56.074 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 podman[100384]: 2024-03-12 04:01:55.91206812 +0000 UTC m=+0.018459077 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:01:56.074 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 podman[100384]: 2024-03-12 04:01:56.012997847 +0000 UTC m=+0.119388787 container create 4c06348cf15a649c115f22eb9223ca1f17958071dd14f90e782003fcfdb1cce1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-12T04:01:56.573 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 podman[100384]: 2024-03-12 04:01:56.224615753 +0000 UTC m=+0.331006713 container init 4c06348cf15a649c115f22eb9223ca1f17958071dd14f90e782003fcfdb1cce1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, ceph=True) 2024-03-12T04:01:56.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 podman[100384]: 2024-03-12 04:01:56.228991234 +0000 UTC m=+0.335382174 container start 4c06348cf15a649c115f22eb9223ca1f17958071dd14f90e782003fcfdb1cce1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, org.label-schema.build-date=20240307, ceph=True, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:01:56.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 ceph-osd[100402]: -- 172.21.15.98:0/966848705 <== mon.1 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562e19e45680 con 0x562e19e0ec00 2024-03-12T04:01:56.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 bash[100384]: 4c06348cf15a649c115f22eb9223ca1f17958071dd14f90e782003fcfdb1cce1 2024-03-12T04:01:56.574 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 systemd[1]: Started Ceph osd.1 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:01:57.073 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:56 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[100398]: 2024-03-12T04:01:56.770+0000 7f9d3c04c7c0 -1 Falling back to public interface 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[88407]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[86021]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-03-12T04:01:57.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:57.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-12T04:01:57.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:57 smithi153 ceph-mon[67960]: pgmap v70: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 117 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-03-12T04:01:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:01:57.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:57 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-12T04:01:57.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:57.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:01:58.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:58 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:58 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:58 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:01:59.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:59 smithi098 ceph-mon[86021]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-12T04:01:59.499 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:01:59 smithi098 ceph-mon[86021]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-03-12T04:01:59.500 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:59 smithi098 ceph-mon[88407]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-12T04:01:59.500 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:01:59 smithi098 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-03-12T04:01:59.500 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[100398]: 2024-03-12T04:01:59.468+0000 7f9d3c04c7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-12T04:01:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:59 smithi153 ceph-mon[67960]: pgmap v71: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-12T04:01:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:01:59 smithi153 ceph-mon[67960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 15 pgs peering) 2024-03-12T04:01:59.764 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:01:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[100398]: 2024-03-12T04:01:59.497+0000 7f9d3c04c7c0 -1 osd.1 118 log_to_monitors true 2024-03-12T04:02:00.383 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:00 smithi098 ceph-mon[86021]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T04:02:00.384 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:00 smithi098 ceph-mon[88407]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T04:02:00.384 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:02:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:02:00] "GET /metrics HTTP/1.1" 200 38230 "" "Prometheus/2.43.0" 2024-03-12T04:02:00.384 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:02:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[100398]: 2024-03-12T04:02:00.259+0000 7f9d2d4a4700 -1 osd.1 118 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:02:00.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:00 smithi153 ceph-mon[67960]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-12T04:02:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[86021]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[86021]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[86021]: osdmap e121: 8 total, 7 up, 8 in 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[86021]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[88407]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[88407]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[88407]: osdmap e121: 8 total, 7 up, 8 in 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[88407]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:01 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:01 smithi153 ceph-mon[67960]: pgmap v72: 161 pgs: 21 active+undersized+degraded, 42 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-12T04:02:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:01 smithi153 ceph-mon[67960]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-12T04:02:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:01 smithi153 ceph-mon[67960]: osdmap e121: 8 total, 7 up, 8 in 2024-03-12T04:02:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:01 smithi153 ceph-mon[67960]: from='osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:01 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:01.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:01 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:02.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[86021]: osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374] boot 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[86021]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[86021]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:02:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[88407]: osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374] boot 2024-03-12T04:02:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[88407]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T04:02:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T04:02:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:02 smithi098 ceph-mon[88407]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T04:02:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:02 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:02:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:02 smithi153 ceph-mon[67960]: osd.1 [v2:172.21.15.98:6810/2074766374,v1:172.21.15.98:6811/2074766374] boot 2024-03-12T04:02:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:02 smithi153 ceph-mon[67960]: osdmap e122: 8 total, 8 up, 8 in 2024-03-12T04:02:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:02 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-12T04:02:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:02 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:02 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:02 smithi153 ceph-mon[67960]: osdmap e123: 8 total, 8 up, 8 in 2024-03-12T04:02:03.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:03 smithi098 ceph-mon[86021]: pgmap v75: 161 pgs: 38 peering, 7 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 42/723 objects degraded (5.809%) 2024-03-12T04:02:03.261 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:03 smithi098 ceph-mon[86021]: Health check update: Degraded data redundancy: 42/723 objects degraded (5.809%), 7 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:03.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:03 smithi098 ceph-mon[88407]: pgmap v75: 161 pgs: 38 peering, 7 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 42/723 objects degraded (5.809%) 2024-03-12T04:02:03.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:03 smithi098 ceph-mon[88407]: Health check update: Degraded data redundancy: 42/723 objects degraded (5.809%), 7 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:03 smithi153 ceph-mon[67960]: pgmap v75: 161 pgs: 38 peering, 7 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 42/723 objects degraded (5.809%) 2024-03-12T04:02:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:03 smithi153 ceph-mon[67960]: Health check update: Degraded data redundancy: 42/723 objects degraded (5.809%), 7 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:05.471 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:05 smithi098 ceph-mon[86021]: pgmap v77: 161 pgs: 38 peering, 7 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/723 objects degraded (5.809%) 2024-03-12T04:02:05.471 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:05 smithi098 ceph-mon[88407]: pgmap v77: 161 pgs: 38 peering, 7 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/723 objects degraded (5.809%) 2024-03-12T04:02:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:05 smithi153 ceph-mon[67960]: pgmap v77: 161 pgs: 38 peering, 7 active+undersized+degraded, 18 active+undersized, 98 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/723 objects degraded (5.809%) 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: Upgrade: Finalizing container_image settings 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:02:06.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: Upgrade: Finalizing container_image settings 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:02:06.827 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:02:06.828 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: Upgrade: Complete! 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.829 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: pgmap v78: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:02:06.830 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: Upgrade: Complete! 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.831 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: pgmap v78: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:06.832 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: Upgrade: Finalizing container_image settings 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:02:06.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.945 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: Upgrade: Complete! 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: pgmap v78: 161 pgs: 38 peering, 123 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:06.946 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:07 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/723 objects degraded (5.809%), 7 pgs degraded) 2024-03-12T04:02:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:07 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:02:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:07 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:07 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/723 objects degraded (5.809%), 7 pgs degraded) 2024-03-12T04:02:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:07 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:02:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:07 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:07 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/723 objects degraded (5.809%), 7 pgs degraded) 2024-03-12T04:02:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:07 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:02:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:07 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:08.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:08 smithi098 ceph-mon[86021]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:02:08.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:08 smithi098 ceph-mon[88407]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:02:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:08 smithi153 ceph-mon[67960]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:02:10.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:02:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:02:10] "GET /metrics HTTP/1.1" 200 38230 "" "Prometheus/2.43.0" 2024-03-12T04:02:11.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:11 smithi098 ceph-mon[86021]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:02:11.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:11 smithi098 ceph-mon[88407]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:02:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:11 smithi153 ceph-mon[67960]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:02:12.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:12.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:12.592 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:12.592 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:13.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:13 smithi098 ceph-mon[88407]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:02:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:13 smithi098 ceph-mon[86021]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:02:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:13 smithi153 ceph-mon[67960]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:02:14.029 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T04:02:14.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:14 smithi098 ceph-mon[86021]: from='client.54288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:14.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:14 smithi098 ceph-mon[88407]: from='client.54288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:14 smithi153 ceph-mon[67960]: from='client.54288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:15.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:15 smithi098 ceph-mon[86021]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-03-12T04:02:15.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:15 smithi098 ceph-mon[88407]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-03-12T04:02:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:15 smithi153 ceph-mon[67960]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 0 op/s 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (4m) 15s ago 26m 27.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (4m) 2m ago 26m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (4m) 15s ago 25m 57.7M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (5m) 2m ago 30m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (15m) 15s ago 32m 504M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (2m) 15s ago 32m 44.8M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (3m) 2m ago 31m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (2m) 15s ago 31m 37.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (5m) 15s ago 27m 14.8M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:02:16.215 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (5m) 2m ago 27m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (55s) 15s ago 30m 42.8M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (20s) 15s ago 30m 12.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (29m) 15s ago 29m 52.5M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (29m) 15s ago 29m 54.8M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (29m) 2m ago 29m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (28m) 2m ago 28m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (28m) 2m ago 28m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (27m) 2m ago 27m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (4m) 2m ago 27m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (25m) 15s ago 25m 78.0M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:02:16.216 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (25m) 2m ago 25m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:02:16.942 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-12T04:02:17.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:17 smithi098 ceph-mon[86021]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:17.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:17 smithi098 ceph-mon[86021]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:17.538 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:17 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:17.538 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:17 smithi098 ceph-mon[88407]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:17.538 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:17 smithi098 ceph-mon[88407]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:17.538 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:17 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:17 smithi153 ceph-mon[67960]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:17 smithi153 ceph-mon[67960]: from='client.64228 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:17 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:18.414 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:18 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:18.414 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:18 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:18.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:18 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:18.907 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:02:19.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:19 smithi098 ceph-mon[86021]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:19.482 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:19 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2646488831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:19.482 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:19 smithi098 ceph-mon[88407]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:19.482 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:19 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2646488831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:19.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-03-12T04:02:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:19 smithi153 ceph-mon[67960]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:19 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2646488831' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:02:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:02:20] "GET /metrics HTTP/1.1" 200 38247 "" "Prometheus/2.43.0" 2024-03-12T04:02:21.295 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:21 smithi098 ceph-mon[86021]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:21.295 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:21 smithi098 ceph-mon[88407]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:21 smithi153 ceph-mon[67960]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:23.136 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:02:23.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:23 smithi098 ceph-mon[86021]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:23.411 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:23 smithi098 ceph-mon[86021]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:23.411 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:23 smithi098 ceph-mon[88407]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:23.411 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:23 smithi098 ceph-mon[88407]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:23 smithi153 ceph-mon[67960]: from='client.54306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:23 smithi153 ceph-mon[67960]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:23.693 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T04:02:25.450 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:25 smithi098 ceph-mon[86021]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:25.451 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:25 smithi098 ceph-mon[88407]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:25.451 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:02:25.451 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": null, 2024-03-12T04:02:25.451 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": false, 2024-03-12T04:02:25.451 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "", 2024-03-12T04:02:25.451 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:02:25.451 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": null, 2024-03-12T04:02:25.452 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T04:02:25.452 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:02:25.452 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:02:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:25 smithi153 ceph-mon[67960]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:26.092 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T04:02:26.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:27.503 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:27 smithi098 ceph-mon[86021]: from='client.64246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:27.504 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:27 smithi098 ceph-mon[86021]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:27.504 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:27 smithi098 ceph-mon[88407]: from='client.64246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:27.504 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:27 smithi098 ceph-mon[88407]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:27.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:27 smithi153 ceph-mon[67960]: from='client.64246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:27.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:27 smithi153 ceph-mon[67960]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:28.230 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T04:02:28.565 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:28 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:28.565 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:28 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/3944562011' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:02:28.566 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:28 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:28.566 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:28 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/3944562011' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:02:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:28 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:28 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/3944562011' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:02:28.897 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-03-12T04:02:29.494 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:29 smithi098 ceph-mon[86021]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:29.495 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:29 smithi098 ceph-mon[88407]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:29 smithi153 ceph-mon[67960]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:30.451 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:02:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:02:30] "GET /metrics HTTP/1.1" 200 38239 "" "Prometheus/2.43.0" 2024-03-12T04:02:31.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:31 smithi098 ceph-mon[86021]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:31.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:31 smithi098 ceph-mon[88407]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:31 smithi153 ceph-mon[67960]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:32.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:32 smithi098 ceph-mon[88407]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:32.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:32 smithi098 ceph-mon[86021]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:32.680 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:32 smithi153 ceph-mon[67960]: from='client.64258 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:33.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T04:02:33.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[88407]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[88407]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[86021]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[86021]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:33 smithi153 ceph-mon[67960]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:33 smithi153 ceph-mon[67960]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:34.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:34.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[86021]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[86021]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[88407]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:34.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:34 smithi098 ceph-mon[88407]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:34 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:34 smithi153 ceph-mon[67960]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:34 smithi153 ceph-mon[67960]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:35.378 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (4m) 35s ago 26m 27.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (4m) 2m ago 26m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (4m) 35s ago 25m 57.7M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (5m) 2m ago 31m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (15m) 35s ago 32m 504M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (2m) 35s ago 33m 44.8M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (3m) 2m ago 31m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (2m) 35s ago 31m 37.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:02:36.036 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (5m) 35s ago 27m 14.8M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (5m) 2m ago 27m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (75s) 35s ago 30m 42.8M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (40s) 35s ago 30m 12.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (30m) 35s ago 30m 52.5M 2992M 17.2.0 e1d6a67b021e da07cdc0934e 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (29m) 35s ago 29m 54.8M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (29m) 2m ago 29m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (29m) 2m ago 29m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (28m) 2m ago 28m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (28m) 2m ago 28m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (5m) 2m ago 27m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (26m) 35s ago 25m 78.0M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:02:36.037 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (25m) 2m ago 25m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:02:36.442 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:02:36.443 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-12T04:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 7 2024-03-12T04:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:02:36.444 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:02:36.782 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-03-12T04:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "2/8 daemons upgraded", 2024-03-12T04:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:02:36.783 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:02:37.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='client.54330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1930972797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='client.54330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-12T04:02:37.576 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:37 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1930972797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='client.54330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-12T04:02:37.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:37 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1930972797' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:38.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:02:38.556 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mgr 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mon 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all crash 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: Upgrade: osd.2 is safe to restart 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:38.557 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mgr 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mon 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all crash 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: Upgrade: osd.2 is safe to restart 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T04:02:38.558 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:38 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:02:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:02:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mgr 2024-03-12T04:02:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mon 2024-03-12T04:02:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all crash 2024-03-12T04:02:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-12T04:02:38.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: Upgrade: osd.2 is safe to restart 2024-03-12T04:02:38.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:02:38.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:38.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:38.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-12T04:02:38.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:38 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:39.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:39 smithi098 ceph-mon[86021]: Upgrade: Updating osd.2 2024-03-12T04:02:39.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:39 smithi098 ceph-mon[86021]: Deploying daemon osd.2 on smithi098 2024-03-12T04:02:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:39 smithi098 ceph-mon[86021]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:39 smithi098 ceph-mon[88407]: Upgrade: Updating osd.2 2024-03-12T04:02:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:39 smithi098 ceph-mon[88407]: Deploying daemon osd.2 on smithi098 2024-03-12T04:02:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:39 smithi098 ceph-mon[88407]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:39 smithi153 ceph-mon[67960]: Upgrade: Updating osd.2 2024-03-12T04:02:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:39 smithi153 ceph-mon[67960]: Deploying daemon osd.2 on smithi098 2024-03-12T04:02:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:39 smithi153 ceph-mon[67960]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:40.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:02:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:02:40] "GET /metrics HTTP/1.1" 200 38239 "" "Prometheus/2.43.0" 2024-03-12T04:02:41.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:41 smithi098 ceph-mon[86021]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:41.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:41 smithi098 ceph-mon[88407]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:41.574 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:41 smithi098 systemd[1]: Stopping Ceph osd.2 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:02:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:41 smithi153 ceph-mon[67960]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:42.073 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[38254]: 2024-03-12T04:02:41.598+0000 7feeeb78f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:02:42.074 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[38254]: 2024-03-12T04:02:41.598+0000 7feeeb78f700 -1 osd.2 123 *** Got signal Terminated *** 2024-03-12T04:02:42.074 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[38254]: 2024-03-12T04:02:41.598+0000 7feeeb78f700 -1 osd.2 123 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:02:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:42 smithi098 ceph-mon[86021]: osd.2 marked itself down and dead 2024-03-12T04:02:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:42 smithi098 ceph-mon[88407]: osd.2 marked itself down and dead 2024-03-12T04:02:42.592 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:42 smithi153 ceph-mon[67960]: osd.2 marked itself down and dead 2024-03-12T04:02:42.892 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:42 smithi098 podman[106518]: 2024-03-12 04:02:42.598386259 +0000 UTC m=+1.118071767 container died da07cdc0934eb7d48ce1938e6e6ad31e1b2c1ffc19dc0c0935c296f3722201a4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, name=centos-stream, GIT_CLEAN=True, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, ceph=True, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, version=8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-12T04:02:43.252 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:42 smithi098 podman[106518]: 2024-03-12 04:02:42.965060113 +0000 UTC m=+1.484745619 container cleanup da07cdc0934eb7d48ce1938e6e6ad31e1b2c1ffc19dc0c0935c296f3722201a4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, vcs-type=git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , name=centos-stream, io.k8s.display-name=CentOS Stream 8, ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-12T04:02:43.252 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:42 smithi098 bash[106518]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2 2024-03-12T04:02:43.568 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:43 smithi098 ceph-mon[86021]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:43.569 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:43 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:02:43.569 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:43 smithi098 ceph-mon[86021]: osdmap e124: 8 total, 7 up, 8 in 2024-03-12T04:02:43.569 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:43 smithi098 ceph-mon[88407]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:43.569 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:43 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:02:43.569 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:43 smithi098 ceph-mon[88407]: osdmap e124: 8 total, 7 up, 8 in 2024-03-12T04:02:43.569 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:43 smithi098 podman[106535]: 2024-03-12 04:02:43.250434284 +0000 UTC m=+0.648420309 container remove da07cdc0934eb7d48ce1938e6e6ad31e1b2c1ffc19dc0c0935c296f3722201a4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, architecture=x86_64, GIT_BRANCH=HEAD, release=754, version=8, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., name=centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base centos centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, maintainer=Guillaume Abrioux , RELEASE=HEAD) 2024-03-12T04:02:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:43 smithi153 ceph-mon[67960]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:43 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:02:43.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:43 smithi153 ceph-mon[67960]: osdmap e124: 8 total, 7 up, 8 in 2024-03-12T04:02:43.824 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:43 smithi098 podman[106628]: 2024-03-12 04:02:43.567309918 +0000 UTC m=+0.101403471 container create 104beb7b2a1bff1dd2f2b48585e3c9db71c8f855a2c9d95cef1176efff853539 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-2b0101e, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-12T04:02:43.824 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:43 smithi098 podman[106628]: 2024-03-12 04:02:43.483502483 +0000 UTC m=+0.017596059 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:43.824 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:43 smithi098 podman[106628]: 2024-03-12 04:02:43.786782766 +0000 UTC m=+0.320876328 container init 104beb7b2a1bff1dd2f2b48585e3c9db71c8f855a2c9d95cef1176efff853539 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T04:02:43.824 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:43 smithi098 podman[106628]: 2024-03-12 04:02:43.789800875 +0000 UTC m=+0.323894430 container start 104beb7b2a1bff1dd2f2b48585e3c9db71c8f855a2c9d95cef1176efff853539 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-12T04:02:44.261 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:43 smithi098 podman[106628]: 2024-03-12 04:02:43.870059986 +0000 UTC m=+0.404153555 container attach 104beb7b2a1bff1dd2f2b48585e3c9db71c8f855a2c9d95cef1176efff853539 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:02:44.261 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:44 smithi098 podman[106628]: 2024-03-12 04:02:44.088911066 +0000 UTC m=+0.623004621 container died 104beb7b2a1bff1dd2f2b48585e3c9db71c8f855a2c9d95cef1176efff853539 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-12T04:02:44.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:44 smithi098 ceph-mon[86021]: osdmap e125: 8 total, 7 up, 8 in 2024-03-12T04:02:44.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:44 smithi098 ceph-mon[88407]: osdmap e125: 8 total, 7 up, 8 in 2024-03-12T04:02:44.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:44 smithi153 ceph-mon[67960]: osdmap e125: 8 total, 7 up, 8 in 2024-03-12T04:02:45.074 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:44 smithi098 podman[106653]: 2024-03-12 04:02:44.706837285 +0000 UTC m=+0.612897805 container remove 104beb7b2a1bff1dd2f2b48585e3c9db71c8f855a2c9d95cef1176efff853539 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-12T04:02:45.074 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:44 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-12T04:02:45.074 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:44 smithi098 systemd[1]: Stopped Ceph osd.2 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:02:45.074 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:44 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.2.service: Consumed 11.845s CPU time. 2024-03-12T04:02:45.074 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:44 smithi098 systemd[1]: Starting Ceph osd.2 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:02:45.405 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-mon[86021]: pgmap v99: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:45.406 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-mon[88407]: pgmap v99: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:45.406 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 podman[106756]: 2024-03-12 04:02:45.052540924 +0000 UTC m=+0.018393830 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:45.406 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 podman[106756]: 2024-03-12 04:02:45.185051084 +0000 UTC m=+0.150903988 container create c53352edd3eb94f66517622c2add82bcb07a81f2ae1273a505541383122a3a49 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-12T04:02:45.406 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 podman[106756]: 2024-03-12 04:02:45.404158126 +0000 UTC m=+0.370011042 container init c53352edd3eb94f66517622c2add82bcb07a81f2ae1273a505541383122a3a49 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T04:02:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:45 smithi153 ceph-mon[67960]: pgmap v99: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:02:45.823 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 podman[106756]: 2024-03-12 04:02:45.407295422 +0000 UTC m=+0.373148325 container start c53352edd3eb94f66517622c2add82bcb07a81f2ae1273a505541383122a3a49 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T04:02:45.823 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 podman[106756]: 2024-03-12 04:02:45.487528889 +0000 UTC m=+0.453381811 container attach c53352edd3eb94f66517622c2add82bcb07a81f2ae1273a505541383122a3a49 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-12T04:02:46.217 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate[106770]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-12T04:02:46.217 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate[106770]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-12T04:02:46.217 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 bash[106756]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-12T04:02:46.217 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 bash[106756]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 bash[106756]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 bash[106756]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 bash[106756]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 bash[106756]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 bash[106756]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate[106770]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate[106770]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate[106770]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate[106770]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate[106770]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-12T04:02:46.218 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:45 smithi098 podman[106756]: 2024-03-12 04:02:45.876421769 +0000 UTC m=+0.842274671 container died c53352edd3eb94f66517622c2add82bcb07a81f2ae1273a505541383122a3a49 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD) 2024-03-12T04:02:46.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:46 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 29/723 objects degraded (4.011%), 4 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:46.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:46 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 29/723 objects degraded (4.011%), 4 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:46.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:46 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 29/723 objects degraded (4.011%), 4 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:46.887 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:46 smithi098 podman[106906]: 2024-03-12 04:02:46.583413926 +0000 UTC m=+0.701780973 container remove c53352edd3eb94f66517622c2add82bcb07a81f2ae1273a505541383122a3a49 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-activate, RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-12T04:02:46.887 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:46 smithi098 podman[106972]: 2024-03-12 04:02:46.77603611 +0000 UTC m=+0.095545225 container create f80bdb9385b3839071b40683bfaff4638da21112e82d38293f846a70b4c677d9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:02:46.887 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:46 smithi098 podman[106972]: 2024-03-12 04:02:46.698767169 +0000 UTC m=+0.018276289 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:02:47.268 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:46 smithi098 podman[106972]: 2024-03-12 04:02:46.970476807 +0000 UTC m=+0.289985923 container init f80bdb9385b3839071b40683bfaff4638da21112e82d38293f846a70b4c677d9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e) 2024-03-12T04:02:47.268 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:46 smithi098 podman[106972]: 2024-03-12 04:02:46.973579742 +0000 UTC m=+0.293088858 container start f80bdb9385b3839071b40683bfaff4638da21112e82d38293f846a70b4c677d9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-12T04:02:47.268 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:46 smithi098 ceph-osd[106991]: -- 172.21.15.98:0/456858071 <== mon.1 v2:172.21.15.98:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de5e55b680 con 0x55de5e524c00 2024-03-12T04:02:47.268 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:47 smithi098 bash[106972]: f80bdb9385b3839071b40683bfaff4638da21112e82d38293f846a70b4c677d9 2024-03-12T04:02:47.268 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:47 smithi098 systemd[1]: Started Ceph osd.2 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:02:47.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[86021]: pgmap v100: 161 pgs: 4 active+undersized+degraded, 9 stale+active+clean, 9 active+undersized, 139 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 29/723 objects degraded (4.011%) 2024-03-12T04:02:47.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.523 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[88407]: pgmap v100: 161 pgs: 4 active+undersized+degraded, 9 stale+active+clean, 9 active+undersized, 139 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 29/723 objects degraded (4.011%) 2024-03-12T04:02:47.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:47 smithi153 ceph-mon[67960]: pgmap v100: 161 pgs: 4 active+undersized+degraded, 9 stale+active+clean, 9 active+undersized, 139 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 29/723 objects degraded (4.011%) 2024-03-12T04:02:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:47.823 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:47 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[106987]: 2024-03-12T04:02:47.521+0000 7f33f958f7c0 -1 Falling back to public interface 2024-03-12T04:02:48.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:48 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:48.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:48 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:48 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:49.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:49 smithi098 ceph-mon[86021]: pgmap v101: 161 pgs: 15 active+undersized+degraded, 31 active+undersized, 115 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 66/723 objects degraded (9.129%) 2024-03-12T04:02:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:49 smithi098 ceph-mon[88407]: pgmap v101: 161 pgs: 15 active+undersized+degraded, 31 active+undersized, 115 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 66/723 objects degraded (9.129%) 2024-03-12T04:02:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:49 smithi153 ceph-mon[67960]: pgmap v101: 161 pgs: 15 active+undersized+degraded, 31 active+undersized, 115 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 66/723 objects degraded (9.129%) 2024-03-12T04:02:50.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-mon[86021]: from='osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T04:02:50.493 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-mon[86021]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T04:02:50.493 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-mon[88407]: from='osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T04:02:50.493 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-mon[88407]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T04:02:50.494 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:02:50] "GET /metrics HTTP/1.1" 200 38246 "" "Prometheus/2.43.0" 2024-03-12T04:02:50.494 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[106987]: 2024-03-12T04:02:50.235+0000 7f33f958f7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-12T04:02:50.494 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[106987]: 2024-03-12T04:02:50.262+0000 7f33f958f7c0 -1 osd.2 123 log_to_monitors true 2024-03-12T04:02:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:50 smithi153 ceph-mon[67960]: from='osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T04:02:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:50 smithi153 ceph-mon[67960]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-12T04:02:51.073 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:02:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[106987]: 2024-03-12T04:02:50.800+0000 7f33e69df700 -1 osd.2 123 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:02:51.471 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[88407]: pgmap v102: 161 pgs: 15 active+undersized+degraded, 31 active+undersized, 115 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 66/723 objects degraded (9.129%) 2024-03-12T04:02:51.471 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[88407]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T04:02:51.471 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[88407]: osdmap e126: 8 total, 7 up, 8 in 2024-03-12T04:02:51.471 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[88407]: from='osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[88407]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[88407]: from='osd.2 ' entity='osd.2' 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[86021]: pgmap v102: 161 pgs: 15 active+undersized+degraded, 31 active+undersized, 115 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 66/723 objects degraded (9.129%) 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[86021]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[86021]: osdmap e126: 8 total, 7 up, 8 in 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[86021]: from='osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[86021]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:51.472 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:51 smithi098 ceph-mon[86021]: from='osd.2 ' entity='osd.2' 2024-03-12T04:02:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:51 smithi153 ceph-mon[67960]: pgmap v102: 161 pgs: 15 active+undersized+degraded, 31 active+undersized, 115 active+clean; 585 KiB data, 138 MiB used, 715 GiB / 715 GiB avail; 66/723 objects degraded (9.129%) 2024-03-12T04:02:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:51 smithi153 ceph-mon[67960]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-12T04:02:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:51 smithi153 ceph-mon[67960]: osdmap e126: 8 total, 7 up, 8 in 2024-03-12T04:02:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:51 smithi153 ceph-mon[67960]: from='osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:51 smithi153 ceph-mon[67960]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:02:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:51 smithi153 ceph-mon[67960]: from='osd.2 ' entity='osd.2' 2024-03-12T04:02:52.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[86021]: osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379] boot 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[86021]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[86021]: pgmap v105: 161 pgs: 24 peering, 6 active+undersized+degraded, 16 active+undersized, 115 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 31/723 objects degraded (4.288%) 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[88407]: osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379] boot 2024-03-12T04:02:52.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[88407]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T04:02:52.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T04:02:52.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:52 smithi098 ceph-mon[88407]: pgmap v105: 161 pgs: 24 peering, 6 active+undersized+degraded, 16 active+undersized, 115 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 31/723 objects degraded (4.288%) 2024-03-12T04:02:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:52 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:52 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:52 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:02:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:52 smithi153 ceph-mon[67960]: osd.2 [v2:172.21.15.98:6818/2437197379,v1:172.21.15.98:6819/2437197379] boot 2024-03-12T04:02:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:52 smithi153 ceph-mon[67960]: osdmap e127: 8 total, 8 up, 8 in 2024-03-12T04:02:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:52 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-12T04:02:52.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:52 smithi153 ceph-mon[67960]: pgmap v105: 161 pgs: 24 peering, 6 active+undersized+degraded, 16 active+undersized, 115 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 31/723 objects degraded (4.288%) 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[86021]: Health check update: Degraded data redundancy: 31/723 objects degraded (4.288%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[86021]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[88407]: Health check update: Degraded data redundancy: 31/723 objects degraded (4.288%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:53 smithi098 ceph-mon[88407]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T04:02:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:53 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:53 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:53 smithi153 ceph-mon[67960]: Health check update: Degraded data redundancy: 31/723 objects degraded (4.288%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T04:02:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:53 smithi153 ceph-mon[67960]: osdmap e128: 8 total, 8 up, 8 in 2024-03-12T04:02:54.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:54 smithi098 ceph-mon[86021]: pgmap v107: 161 pgs: 24 peering, 6 active+undersized+degraded, 16 active+undersized, 115 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/723 objects degraded (4.288%) 2024-03-12T04:02:54.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:54 smithi098 ceph-mon[88407]: pgmap v107: 161 pgs: 24 peering, 6 active+undersized+degraded, 16 active+undersized, 115 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/723 objects degraded (4.288%) 2024-03-12T04:02:54.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:54 smithi153 ceph-mon[67960]: pgmap v107: 161 pgs: 24 peering, 6 active+undersized+degraded, 16 active+undersized, 115 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 31/723 objects degraded (4.288%) 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: pgmap v108: 161 pgs: 24 peering, 137 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: Upgrade: Finalizing container_image settings 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:57.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: pgmap v108: 161 pgs: 24 peering, 137 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:57.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: Upgrade: Finalizing container_image settings 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:02:57.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:02:57.328 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.329 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: Upgrade: Complete! 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:02:57.330 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.331 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: Upgrade: Complete! 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.332 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: pgmap v108: 161 pgs: 24 peering, 137 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:02:57.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: Upgrade: Finalizing container_image settings 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:02:57.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: Upgrade: Complete! 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:02:57.446 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:02:58.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:58 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/723 objects degraded (4.288%), 6 pgs degraded) 2024-03-12T04:02:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:58 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:02:58.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:58 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:58 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/723 objects degraded (4.288%), 6 pgs degraded) 2024-03-12T04:02:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:58 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:02:58.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:58 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:58 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 31/723 objects degraded (4.288%), 6 pgs degraded) 2024-03-12T04:02:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:58 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:02:58.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:58 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:02:59.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:02:59 smithi098 ceph-mon[86021]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:02:59.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:02:59 smithi098 ceph-mon[88407]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:02:59.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:02:59 smithi153 ceph-mon[67960]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:03:00.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:03:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:03:00] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.43.0" 2024-03-12T04:03:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:01 smithi098 ceph-mon[86021]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:01.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:01 smithi098 ceph-mon[88407]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:01 smithi153 ceph-mon[67960]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:02.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:02 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:02 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:02 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:03 smithi098 ceph-mon[86021]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:03:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:03 smithi098 ceph-mon[88407]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:03:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:03 smithi153 ceph-mon[67960]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:03:05.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:05 smithi098 ceph-mon[86021]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-12T04:03:05.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:05 smithi098 ceph-mon[88407]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-12T04:03:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:05 smithi153 ceph-mon[67960]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-12T04:03:07.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:07 smithi098 ceph-mon[86021]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:07.425 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:07.425 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:07 smithi098 ceph-mon[88407]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:07.425 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:07 smithi153 ceph-mon[67960]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:07.706 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T04:03:08.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:08 smithi098 ceph-mon[86021]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:08.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:08 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:08.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:08 smithi098 ceph-mon[88407]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:08.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:08 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:08 smithi153 ceph-mon[67960]: from='client.54366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:08 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:09.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:09 smithi098 ceph-mon[86021]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:09.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:09 smithi098 ceph-mon[88407]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:09 smithi153 ceph-mon[67960]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:09.716 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:03:09.716 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (5m) 18s ago 27m 27.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (5m) 3m ago 26m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (4m) 18s ago 26m 59.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (6m) 3m ago 31m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (16m) 18s ago 33m 506M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (3m) 18s ago 33m 47.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (4m) 3m ago 32m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (2m) 18s ago 32m 40.4M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (6m) 18s ago 28m 15.2M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (6m) 3m ago 28m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (108s) 18s ago 31m 43.4M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (73s) 18s ago 31m 44.1M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (23s) 18s ago 30m 12.7M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:03:09.717 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (30m) 18s ago 30m 56.1M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:03:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (29m) 3m ago 29m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:03:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (29m) 3m ago 29m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:03:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (29m) 3m ago 29m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:03:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (28m) 3m ago 28m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:03:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (5m) 3m ago 28m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:03:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (26m) 18s ago 26m 78.2M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:03:09.718 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (26m) 3m ago 26m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:03:10.540 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:03:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:03:10] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.43.0" 2024-03-12T04:03:10.564 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-12T04:03:11.517 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[86021]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:11.518 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[86021]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:11.518 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:11.518 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:11.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[88407]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:11.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[88407]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:11.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:11.518 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:11 smithi153 ceph-mon[67960]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:11 smithi153 ceph-mon[67960]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:12.818 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:03:13.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:13 smithi098 ceph-mon[86021]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:13.556 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:13 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1418585827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:13.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:13 smithi098 ceph-mon[88407]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:13.557 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:13 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1418585827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:13.579 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-03-12T04:03:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:13 smithi153 ceph-mon[67960]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:13 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1418585827' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:15.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:15 smithi098 ceph-mon[86021]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:15.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:15 smithi098 ceph-mon[88407]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:15 smithi153 ceph-mon[67960]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:17.067 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:03:17.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:17 smithi098 ceph-mon[88407]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:17.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:17 smithi098 ceph-mon[88407]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:17 smithi098 ceph-mon[86021]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:17 smithi098 ceph-mon[86021]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:17 smithi153 ceph-mon[67960]: from='client.64318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:17 smithi153 ceph-mon[67960]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:17.780 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T04:03:18.342 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:18 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:18.343 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:18 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:18 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:19 smithi098 ceph-mon[86021]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:19 smithi098 ceph-mon[88407]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:19 smithi153 ceph-mon[67960]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:19.734 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:03:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": null, 2024-03-12T04:03:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": false, 2024-03-12T04:03:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "", 2024-03-12T04:03:19.734 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:03:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": null, 2024-03-12T04:03:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T04:03:19.735 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:03:19.735 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:03:20.341 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T04:03:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:03:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:03:20] "GET /metrics HTTP/1.1" 200 38254 "" "Prometheus/2.43.0" 2024-03-12T04:03:21.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:21 smithi098 ceph-mon[86021]: from='client.54390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:21.524 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:21 smithi098 ceph-mon[86021]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:21.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:21 smithi098 ceph-mon[88407]: from='client.54390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:21.524 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:21 smithi098 ceph-mon[88407]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:21 smithi153 ceph-mon[67960]: from='client.54390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:21 smithi153 ceph-mon[67960]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:22.358 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T04:03:22.981 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-03-12T04:03:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:23 smithi098 ceph-mon[86021]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:23.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:23 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1935796111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:03:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:23 smithi098 ceph-mon[88407]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:23 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1935796111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:03:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:23 smithi153 ceph-mon[67960]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:23 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1935796111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:03:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:25 smithi153 ceph-mon[67960]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:25.706 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:25 smithi098 ceph-mon[86021]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:25.706 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:25 smithi098 ceph-mon[88407]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:26 smithi153 ceph-mon[67960]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:26 smithi153 ceph-mon[67960]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:26.766 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:26 smithi098 ceph-mon[86021]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:26.766 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:26 smithi098 ceph-mon[86021]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:26.766 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:26.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:26 smithi098 ceph-mon[88407]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:26.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:26 smithi098 ceph-mon[88407]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:26.767 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:26.839 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:27.480 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T04:03:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[86021]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:03:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[86021]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:28.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:28.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[88407]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:28.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:28.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:28.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:03:28.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:28.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[88407]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:28.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:27 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:27 smithi153 ceph-mon[67960]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:03:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:27 smithi153 ceph-mon[67960]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:28.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:27 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:29.014 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:28 smithi098 ceph-mon[86021]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:29.014 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:28 smithi098 ceph-mon[88407]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:29.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:28 smithi153 ceph-mon[67960]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:30.049 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mgr 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mon 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.050 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all crash 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[86021]: Upgrade: osd.3 is safe to restart 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mgr 2024-03-12T04:03:30.051 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mon 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all crash 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-12T04:03:30.052 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:29 smithi098 ceph-mon[88407]: Upgrade: osd.3 is safe to restart 2024-03-12T04:03:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:03:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:03:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:03:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mgr 2024-03-12T04:03:30.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mon 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all crash 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-12T04:03:30.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:29 smithi153 ceph-mon[67960]: Upgrade: osd.3 is safe to restart 2024-03-12T04:03:30.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:03:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:03:30] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.43.0" 2024-03-12T04:03:30.512 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:03:30.978 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:30 smithi098 ceph-mon[86021]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:30.979 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:30 smithi098 ceph-mon[88407]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:31.175 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:03:31.175 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (5m) 39s ago 27m 27.7M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:03:31.175 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (5m) 3m ago 27m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:03:31.175 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (5m) 39s ago 26m 59.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:03:31.175 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (6m) 3m ago 32m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (16m) 39s ago 33m 506M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (3m) 39s ago 33m 47.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (4m) 3m ago 32m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (3m) 39s ago 32m 40.4M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (6m) 39s ago 28m 15.2M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (6m) 3m ago 28m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (2m) 39s ago 31m 43.4M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (95s) 39s ago 31m 44.1M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (44s) 39s ago 31m 12.7M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (30m) 39s ago 30m 56.1M 2992M 17.2.0 e1d6a67b021e 05f1eb9a83ff 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (30m) 3m ago 30m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (29m) 3m ago 29m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:03:31.176 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (29m) 3m ago 29m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:03:31.177 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (29m) 3m ago 29m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:03:31.177 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (6m) 3m ago 28m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:03:31.177 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (26m) 39s ago 26m 78.2M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:03:31.177 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (26m) 3m ago 26m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:03:31.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:30 smithi153 ceph-mon[67960]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:03:31.577 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:03:31.578 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:03:31.578 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:03:31.578 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-12T04:03:31.578 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 8 2024-03-12T04:03:31.578 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:03:31.578 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:03:31.942 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:03:31.942 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:03:31.942 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:03:31.942 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-12T04:03:31.943 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:03:31.943 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "3/8 daemons upgraded", 2024-03-12T04:03:31.943 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:03:31.943 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:03:31.943 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:03:32.136 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[86021]: from='client.54408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.136 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[86021]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[86021]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/3441438640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[88407]: from='client.54408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[88407]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[88407]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T04:03:32.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:32.138 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:31 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/3441438640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:31 smithi153 ceph-mon[67960]: from='client.54408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:31 smithi153 ceph-mon[67960]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:31 smithi153 ceph-mon[67960]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:31 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:31 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-12T04:03:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:31 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:32.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:31 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/3441438640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:32 smithi153 ceph-mon[67960]: Upgrade: Updating osd.3 2024-03-12T04:03:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:32 smithi153 ceph-mon[67960]: Deploying daemon osd.3 on smithi098 2024-03-12T04:03:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:32 smithi153 ceph-mon[67960]: from='client.54432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:33.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:32 smithi153 ceph-mon[67960]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:33.315 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[86021]: Upgrade: Updating osd.3 2024-03-12T04:03:33.316 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[86021]: Deploying daemon osd.3 on smithi098 2024-03-12T04:03:33.316 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[86021]: from='client.54432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:33.316 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[86021]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:33.316 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[88407]: Upgrade: Updating osd.3 2024-03-12T04:03:33.316 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[88407]: Deploying daemon osd.3 on smithi098 2024-03-12T04:03:33.316 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[88407]: from='client.54432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:03:33.316 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:32 smithi098 ceph-mon[88407]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:03:35.300 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:35 smithi098 ceph-mon[86021]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:35.301 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:35 smithi098 ceph-mon[88407]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:35.301 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:35 smithi098 systemd[1]: Stopping Ceph osd.3 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:03:35.573 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[41222]: 2024-03-12T04:03:35.297+0000 7fb0281fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:03:35.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[41222]: 2024-03-12T04:03:35.297+0000 7fb0281fa700 -1 osd.3 128 *** Got signal Terminated *** 2024-03-12T04:03:35.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:35 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[41222]: 2024-03-12T04:03:35.297+0000 7fb0281fa700 -1 osd.3 128 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:03:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:35 smithi153 ceph-mon[67960]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:36.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:36 smithi098 ceph-mon[86021]: osd.3 marked itself down and dead 2024-03-12T04:03:36.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:36 smithi098 ceph-mon[88407]: osd.3 marked itself down and dead 2024-03-12T04:03:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:36 smithi153 ceph-mon[67960]: osd.3 marked itself down and dead 2024-03-12T04:03:37.027 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:36 smithi098 podman[113014]: 2024-03-12 04:03:36.709164849 +0000 UTC m=+1.525304196 container died 05f1eb9a83ff16ce4cbbc9279aa9b6cb29840ada57ad32dd6112aaccec180425 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, name=centos-stream, release=754, version=8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, vendor=Red Hat, Inc.) 2024-03-12T04:03:37.311 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:37 smithi098 ceph-mon[88407]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:37.311 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:37 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:03:37.311 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:37 smithi098 ceph-mon[88407]: osdmap e129: 8 total, 7 up, 8 in 2024-03-12T04:03:37.311 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:37 smithi098 ceph-mon[86021]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:37.311 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:37 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:03:37.311 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:37 smithi098 ceph-mon[86021]: osdmap e129: 8 total, 7 up, 8 in 2024-03-12T04:03:37.312 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:37 smithi098 podman[113014]: 2024-03-12 04:03:37.248217016 +0000 UTC m=+2.064356418 container cleanup 05f1eb9a83ff16ce4cbbc9279aa9b6cb29840ada57ad32dd6112aaccec180425 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=-17.2.0, ceph=True, io.openshift.expose-services=, GIT_BRANCH=HEAD, distribution-scope=public, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, release=754, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, maintainer=Guillaume Abrioux , RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vendor=Red Hat, Inc.) 2024-03-12T04:03:37.312 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:37 smithi098 bash[113014]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3 2024-03-12T04:03:37.573 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:37 smithi098 podman[113031]: 2024-03-12 04:03:37.500276215 +0000 UTC m=+0.785690611 container remove 05f1eb9a83ff16ce4cbbc9279aa9b6cb29840ada57ad32dd6112aaccec180425 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-type=git, maintainer=Guillaume Abrioux , distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, version=8, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-12T04:03:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:37 smithi153 ceph-mon[67960]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:03:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:37 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:03:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:37 smithi153 ceph-mon[67960]: osdmap e129: 8 total, 7 up, 8 in 2024-03-12T04:03:38.264 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:37 smithi098 podman[113124]: 2024-03-12 04:03:37.822278413 +0000 UTC m=+0.017568153 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:38.265 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:37 smithi098 podman[113124]: 2024-03-12 04:03:37.959494895 +0000 UTC m=+0.154784635 container create 0ec5c4e6f9c9295f1535091e9f7f1be3eeaba77e80e9fd50970ceed9a239a638 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2) 2024-03-12T04:03:38.265 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:38 smithi098 podman[113124]: 2024-03-12 04:03:38.161814963 +0000 UTC m=+0.357104703 container init 0ec5c4e6f9c9295f1535091e9f7f1be3eeaba77e80e9fd50970ceed9a239a638 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:03:38.265 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:38 smithi098 podman[113124]: 2024-03-12 04:03:38.164871846 +0000 UTC m=+0.360161586 container start 0ec5c4e6f9c9295f1535091e9f7f1be3eeaba77e80e9fd50970ceed9a239a638 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=main-2b0101e) 2024-03-12T04:03:38.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:38 smithi098 ceph-mon[86021]: osdmap e130: 8 total, 7 up, 8 in 2024-03-12T04:03:38.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:38 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:38.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:38 smithi098 ceph-mon[88407]: osdmap e130: 8 total, 7 up, 8 in 2024-03-12T04:03:38.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:38 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:38.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:38 smithi098 podman[113124]: 2024-03-12 04:03:38.261932239 +0000 UTC m=+0.457221982 container attach 0ec5c4e6f9c9295f1535091e9f7f1be3eeaba77e80e9fd50970ceed9a239a638 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-12T04:03:38.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:38 smithi098 podman[113124]: 2024-03-12 04:03:38.463479423 +0000 UTC m=+0.658769181 container died 0ec5c4e6f9c9295f1535091e9f7f1be3eeaba77e80e9fd50970ceed9a239a638 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-12T04:03:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:38 smithi153 ceph-mon[67960]: osdmap e130: 8 total, 7 up, 8 in 2024-03-12T04:03:38.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:38 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:39.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:39 smithi098 ceph-mon[86021]: pgmap v131: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:39 smithi098 ceph-mon[86021]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-12T04:03:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:39 smithi098 ceph-mon[88407]: pgmap v131: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:39 smithi098 ceph-mon[88407]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-12T04:03:39.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 podman[113150]: 2024-03-12 04:03:39.149366479 +0000 UTC m=+0.679491853 container remove 0ec5c4e6f9c9295f1535091e9f7f1be3eeaba77e80e9fd50970ceed9a239a638 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:03:39.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-12T04:03:39.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 systemd[1]: Stopped Ceph osd.3 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:03:39.575 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.3.service: Consumed 16.187s CPU time. 2024-03-12T04:03:39.575 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 systemd[1]: Starting Ceph osd.3 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:03:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:39 smithi153 ceph-mon[67960]: pgmap v131: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:39 smithi153 ceph-mon[67960]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-03-12T04:03:39.850 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 podman[113257]: 2024-03-12 04:03:39.49708585 +0000 UTC m=+0.018411422 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:39.850 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 podman[113257]: 2024-03-12 04:03:39.742424352 +0000 UTC m=+0.263749982 container create 68001b60222c9bbc0f9b1c0ea07786bebe604cbaace57dd76ad1d8fc72ba0fe9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-12T04:03:40.219 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 podman[113257]: 2024-03-12 04:03:39.928304451 +0000 UTC m=+0.449630034 container init 68001b60222c9bbc0f9b1c0ea07786bebe604cbaace57dd76ad1d8fc72ba0fe9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T04:03:40.220 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:39 smithi098 podman[113257]: 2024-03-12 04:03:39.931375928 +0000 UTC m=+0.452701499 container start 68001b60222c9bbc0f9b1c0ea07786bebe604cbaace57dd76ad1d8fc72ba0fe9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-12T04:03:40.220 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 podman[113257]: 2024-03-12 04:03:40.003092238 +0000 UTC m=+0.524417823 container attach 68001b60222c9bbc0f9b1c0ea07786bebe604cbaace57dd76ad1d8fc72ba0fe9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-12T04:03:40.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:03:40] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.43.0" 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate[113272]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate[113272]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 bash[113257]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 bash[113257]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 bash[113257]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 bash[113257]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 bash[113257]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 bash[113257]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 bash[113257]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate[113272]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate[113272]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T04:03:40.574 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate[113272]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-12T04:03:40.575 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate[113272]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-12T04:03:40.575 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate[113272]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-12T04:03:40.575 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:40 smithi098 podman[113257]: 2024-03-12 04:03:40.397401433 +0000 UTC m=+0.918726992 container died 68001b60222c9bbc0f9b1c0ea07786bebe604cbaace57dd76ad1d8fc72ba0fe9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T04:03:41.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:41 smithi098 ceph-mon[86021]: pgmap v132: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:41.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:41.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:41 smithi098 ceph-mon[88407]: pgmap v132: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:41.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:41.324 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 podman[113402]: 2024-03-12 04:03:41.048587455 +0000 UTC m=+0.645503831 container remove 68001b60222c9bbc0f9b1c0ea07786bebe604cbaace57dd76ad1d8fc72ba0fe9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:03:41.324 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 podman[113466]: 2024-03-12 04:03:41.165809444 +0000 UTC m=+0.019085424 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:03:41.324 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 podman[113466]: 2024-03-12 04:03:41.291352631 +0000 UTC m=+0.144628648 container create b8b05419735b77dbe9fedd26503139cdf6ee3e4157e772d9f515309054ed15cb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-12T04:03:41.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:41 smithi153 ceph-mon[67960]: pgmap v132: 161 pgs: 32 peering, 13 stale+active+clean, 116 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:41.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:41.694 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:41.823 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 podman[113466]: 2024-03-12 04:03:41.535762439 +0000 UTC m=+0.389038427 container init b8b05419735b77dbe9fedd26503139cdf6ee3e4157e772d9f515309054ed15cb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T04:03:41.823 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 podman[113466]: 2024-03-12 04:03:41.538840322 +0000 UTC m=+0.392116298 container start b8b05419735b77dbe9fedd26503139cdf6ee3e4157e772d9f515309054ed15cb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, ceph=True, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T04:03:41.824 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 ceph-osd[113486]: -- 172.21.15.98:0/879044782 <== mon.0 v2:172.21.15.98:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55db4911f680 con 0x55db490e8c00 2024-03-12T04:03:41.824 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 bash[113466]: b8b05419735b77dbe9fedd26503139cdf6ee3e4157e772d9f515309054ed15cb 2024-03-12T04:03:41.824 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:41 smithi098 systemd[1]: Started Ceph osd.3 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:03:42.510 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:42 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[113482]: 2024-03-12T04:03:42.087+0000 7f538835a7c0 -1 Falling back to public interface 2024-03-12T04:03:42.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:42.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:42.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:42 smithi098 ceph-mon[86021]: pgmap v133: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:42 smithi098 ceph-mon[88407]: pgmap v133: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:42.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:42.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:42.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:42 smithi153 ceph-mon[67960]: pgmap v133: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:43.711 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:43 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T04:03:43.711 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:43 smithi098 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 13 pgs peering) 2024-03-12T04:03:43.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:43 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T04:03:43.711 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:43 smithi098 ceph-mon[86021]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 13 pgs peering) 2024-03-12T04:03:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:43 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 27 pgs degraded (PG_DEGRADED) 2024-03-12T04:03:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:43 smithi153 ceph-mon[67960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 13 pgs peering) 2024-03-12T04:03:45.009 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:44 smithi098 ceph-mon[86021]: pgmap v134: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:45.010 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:44 smithi098 ceph-mon[88407]: pgmap v134: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:45.010 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[113482]: 2024-03-12T04:03:44.785+0000 7f538835a7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-12T04:03:45.010 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[113482]: 2024-03-12T04:03:44.813+0000 7f538835a7c0 -1 osd.3 128 log_to_monitors true 2024-03-12T04:03:45.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:44 smithi153 ceph-mon[67960]: pgmap v134: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:45.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:45 smithi098 ceph-mon[86021]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T04:03:45.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:45 smithi098 ceph-mon[88407]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T04:03:46.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:45 smithi153 ceph-mon[67960]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-12T04:03:46.573 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[113482]: 2024-03-12T04:03:46.177+0000 7f53757aa700 -1 osd.3 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:03:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[86021]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T04:03:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[86021]: osdmap e131: 8 total, 7 up, 8 in 2024-03-12T04:03:47.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[86021]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[86021]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[86021]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[88407]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[88407]: osdmap e131: 8 total, 7 up, 8 in 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[88407]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[88407]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' 2024-03-12T04:03:47.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:46 smithi098 ceph-mon[88407]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:46 smithi153 ceph-mon[67960]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-12T04:03:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:46 smithi153 ceph-mon[67960]: osdmap e131: 8 total, 7 up, 8 in 2024-03-12T04:03:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:46 smithi153 ceph-mon[67960]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi098", "root=default"]}]: dispatch 2024-03-12T04:03:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:46 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:46 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:46 smithi153 ceph-mon[67960]: from='osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884]' entity='osd.3' 2024-03-12T04:03:47.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:46 smithi153 ceph-mon[67960]: pgmap v136: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-12T04:03:47.994 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[86021]: osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884] boot 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[86021]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[88407]: osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884] boot 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[88407]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.995 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:47.996 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:47 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:47 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:03:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:47 smithi153 ceph-mon[67960]: osd.3 [v2:172.21.15.98:6826/2562679884,v1:172.21.15.98:6827/2562679884] boot 2024-03-12T04:03:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:47 smithi153 ceph-mon[67960]: osdmap e132: 8 total, 8 up, 8 in 2024-03-12T04:03:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-12T04:03:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:48.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:47 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:49 smithi153 ceph-mon[67960]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T04:03:49.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:49 smithi153 ceph-mon[67960]: pgmap v139: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-03-12T04:03:49.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:49 smithi098 ceph-mon[86021]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T04:03:49.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:49 smithi098 ceph-mon[86021]: pgmap v139: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-03-12T04:03:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:49 smithi098 ceph-mon[88407]: osdmap e133: 8 total, 8 up, 8 in 2024-03-12T04:03:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:49 smithi098 ceph-mon[88407]: pgmap v139: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-03-12T04:03:50.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:03:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:03:50] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.43.0" 2024-03-12T04:03:51.253 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:03:51.133+0000 7f79ceb0e700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:03:51.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: pgmap v140: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:03:51.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: pgmap v140: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:03:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: pgmap v140: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-03-12T04:03:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:03:51.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:03:52.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:52 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:03:52.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:52 smithi098 ceph-mon[86021]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:03:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:52 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:03:52.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:52 smithi098 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:03:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:52 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:03:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:52 smithi153 ceph-mon[67960]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-03-12T04:03:53.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:53 smithi098 ceph-mon[86021]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:53 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/723 objects degraded (13.140%), 27 pgs degraded) 2024-03-12T04:03:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:53 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:03:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:53 smithi098 ceph-mon[88407]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:53 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/723 objects degraded (13.140%), 27 pgs degraded) 2024-03-12T04:03:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:53 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:03:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:53 smithi153 ceph-mon[67960]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:53 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/723 objects degraded (13.140%), 27 pgs degraded) 2024-03-12T04:03:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:53 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:03:55.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:55 smithi098 ceph-mon[86021]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:55.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:55 smithi098 ceph-mon[88407]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:55 smithi153 ceph-mon[67960]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:57.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:57 smithi098 ceph-mon[86021]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:57.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:57 smithi098 ceph-mon[88407]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:57 smithi153 ceph-mon[67960]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:03:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:03:58.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:58 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:58.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:58 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:58 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:03:59.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:03:59 smithi098 ceph-mon[86021]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:59.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:03:59 smithi098 ceph-mon[88407]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:03:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:03:59 smithi153 ceph-mon[67960]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:04:00.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:04:00] "GET /metrics HTTP/1.1" 200 38258 "" "Prometheus/2.43.0" 2024-03-12T04:04:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:01 smithi098 ceph-mon[86021]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:04:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:01 smithi098 ceph-mon[88407]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:04:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:01 smithi153 ceph-mon[67960]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:04:02.306 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:04:02.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:02 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:02 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:02 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:02.952 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:04:02.952 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (6m) 16s ago 28m 27.6M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:04:02.952 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (6m) 4m ago 27m 80.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:04:02.952 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (5m) 16s ago 27m 62.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:04:02.952 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (7m) 4m ago 32m 447M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:04:02.952 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (17m) 16s ago 34m 508M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:04:02.952 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (4m) 16s ago 34m 50.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (5m) 4m ago 32m 37.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (3m) 16s ago 33m 42.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (7m) 16s ago 28m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (6m) 4m ago 28m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (2m) 16s ago 32m 44.9M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (2m) 16s ago 32m 44.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (76s) 16s ago 31m 49.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (21s) 16s ago 31m 11.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (30m) 4m ago 30m 59.6M 3248M 17.2.0 e1d6a67b021e 14db0b207998 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (30m) 4m ago 30m 55.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (30m) 4m ago 30m 53.0M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (29m) 4m ago 29m 55.7M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:04:02.953 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (6m) 4m ago 29m 68.0M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:04:02.954 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (27m) 16s ago 27m 78.4M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:04:02.954 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (27m) 4m ago 27m 80.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 4 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:03.340 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:04:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:04:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:04:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-12T04:04:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 9 2024-03-12T04:04:03.341 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:04:03.341 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:04:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:03 smithi098 ceph-mon[86021]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T04:04:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:03 smithi098 ceph-mon[86021]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:03 smithi098 ceph-mon[88407]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T04:04:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:03 smithi098 ceph-mon[88407]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "4/8 daemons upgraded", 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:04:03.668 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:04:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:03 smithi153 ceph-mon[67960]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-12T04:04:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:03 smithi153 ceph-mon[67960]: from='client.64378 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:04.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:04 smithi153 ceph-mon[67960]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:04.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:04 smithi153 ceph-mon[67960]: from='client.64390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:04.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:04 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1228532579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:04.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:04 smithi098 ceph-mon[86021]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:04.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:04 smithi098 ceph-mon[86021]: from='client.64390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:04.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:04 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1228532579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:04.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:04 smithi098 ceph-mon[88407]: from='client.54450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:04.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:04 smithi098 ceph-mon[88407]: from='client.64390 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:04.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:04 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1228532579' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:05 smithi153 ceph-mon[67960]: from='client.54468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:05 smithi153 ceph-mon[67960]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:05.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:05 smithi098 ceph-mon[88407]: from='client.54468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:05.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:05 smithi098 ceph-mon[88407]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:05.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:05 smithi098 ceph-mon[86021]: from='client.54468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:05.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:05 smithi098 ceph-mon[86021]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:06.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:04:06.826 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:04:06.826 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:04:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:07 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:04:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:07 smithi153 ceph-mon[67960]: Upgrade: osd.4 is safe to restart 2024-03-12T04:04:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:07 smithi153 ceph-mon[67960]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:07.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:07 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:04:07.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:07 smithi098 ceph-mon[88407]: Upgrade: osd.4 is safe to restart 2024-03-12T04:04:07.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:07 smithi098 ceph-mon[88407]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:07 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-12T04:04:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:07 smithi098 ceph-mon[86021]: Upgrade: osd.4 is safe to restart 2024-03-12T04:04:07.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:07 smithi098 ceph-mon[86021]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:08 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:08 smithi153 ceph-mon[67960]: Upgrade: Updating osd.4 2024-03-12T04:04:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T04:04:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:08.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:08 smithi153 ceph-mon[67960]: Deploying daemon osd.4 on smithi153 2024-03-12T04:04:08.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:08 smithi153 ceph-mon[67960]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:09.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:09.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[88407]: Upgrade: Updating osd.4 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[88407]: Deploying daemon osd.4 on smithi153 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[88407]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[86021]: Upgrade: Updating osd.4 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[86021]: Deploying daemon osd.4 on smithi153 2024-03-12T04:04:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:08 smithi098 ceph-mon[86021]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:10.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:04:10] "GET /metrics HTTP/1.1" 200 38258 "" "Prometheus/2.43.0" 2024-03-12T04:04:11.153 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:10 smithi153 systemd[1]: Stopping Ceph osd.4 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:04:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:11 smithi153 ceph-mon[67960]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:11 smithi153 ceph-mon[67960]: osd.4 marked itself down and dead 2024-03-12T04:04:11.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:11.443 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[29021]: 2024-03-12T04:04:11.150+0000 7ff4f3786700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:04:11.443 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[29021]: 2024-03-12T04:04:11.150+0000 7ff4f3786700 -1 osd.4 133 *** Got signal Terminated *** 2024-03-12T04:04:11.443 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:11 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[29021]: 2024-03-12T04:04:11.150+0000 7ff4f3786700 -1 osd.4 133 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:04:11.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:11 smithi098 ceph-mon[86021]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:11.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:11 smithi098 ceph-mon[86021]: osd.4 marked itself down and dead 2024-03-12T04:04:11.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:11 smithi098 ceph-mon[88407]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:11 smithi098 ceph-mon[88407]: osd.4 marked itself down and dead 2024-03-12T04:04:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:11.911 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:11 smithi153 podman[77635]: 2024-03-12 04:04:11.640839525 +0000 UTC m=+0.600257450 container died 14db0b2079984f76f747954351afda219ed32bee0ad9d3468103267c344aba35 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, release=754, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, name=centos-stream, io.openshift.tags=base centos centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vendor=Red Hat, Inc.) 2024-03-12T04:04:12.192 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 podman[77635]: 2024-03-12 04:04:12.000581641 +0000 UTC m=+0.959999567 container cleanup 14db0b2079984f76f747954351afda219ed32bee0ad9d3468103267c344aba35 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., vcs-type=git, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-12T04:04:12.192 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 bash[77635]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4 2024-03-12T04:04:12.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:12 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:04:12.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:12 smithi098 ceph-mon[86021]: osdmap e134: 8 total, 7 up, 8 in 2024-03-12T04:04:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:12 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:04:12.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:12 smithi098 ceph-mon[88407]: osdmap e134: 8 total, 7 up, 8 in 2024-03-12T04:04:12.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:12 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:04:12.612 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:12 smithi153 ceph-mon[67960]: osdmap e134: 8 total, 7 up, 8 in 2024-03-12T04:04:12.612 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 podman[77652]: 2024-03-12 04:04:12.277627144 +0000 UTC m=+0.638304475 container remove 14db0b2079984f76f747954351afda219ed32bee0ad9d3468103267c344aba35 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.tags=base centos centos-stream, distribution-scope=public, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, ceph=True, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870) 2024-03-12T04:04:12.915 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 podman[77745]: 2024-03-12 04:04:12.610923512 +0000 UTC m=+0.114796436 container create bb21ea95d88a393c36948d5b07498a0a5cb9b350cc49567b12bf75ce7b53dc7a (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-12T04:04:12.915 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 podman[77745]: 2024-03-12 04:04:12.515212478 +0000 UTC m=+0.019085406 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:04:12.915 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 podman[77745]: 2024-03-12 04:04:12.813279803 +0000 UTC m=+0.317152727 container init bb21ea95d88a393c36948d5b07498a0a5cb9b350cc49567b12bf75ce7b53dc7a (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T04:04:12.915 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 podman[77745]: 2024-03-12 04:04:12.816842172 +0000 UTC m=+0.320715097 container start bb21ea95d88a393c36948d5b07498a0a5cb9b350cc49567b12bf75ce7b53dc7a (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-03-12T04:04:12.915 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:12 smithi153 podman[77745]: 2024-03-12 04:04:12.913235713 +0000 UTC m=+0.417108642 container attach bb21ea95d88a393c36948d5b07498a0a5cb9b350cc49567b12bf75ce7b53dc7a (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-12T04:04:13.192 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:13 smithi153 podman[77745]: 2024-03-12 04:04:13.116141667 +0000 UTC m=+0.620014601 container died bb21ea95d88a393c36948d5b07498a0a5cb9b350cc49567b12bf75ce7b53dc7a (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T04:04:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:13 smithi098 ceph-mon[86021]: pgmap v152: 161 pgs: 33 peering, 14 stale+active+clean, 114 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T04:04:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:13 smithi098 ceph-mon[86021]: Health check failed: Reduced data availability: 6 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-12T04:04:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:13 smithi098 ceph-mon[86021]: osdmap e135: 8 total, 7 up, 8 in 2024-03-12T04:04:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:13 smithi098 ceph-mon[88407]: pgmap v152: 161 pgs: 33 peering, 14 stale+active+clean, 114 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T04:04:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:13 smithi098 ceph-mon[88407]: Health check failed: Reduced data availability: 6 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-12T04:04:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:13 smithi098 ceph-mon[88407]: osdmap e135: 8 total, 7 up, 8 in 2024-03-12T04:04:13.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:13 smithi153 ceph-mon[67960]: pgmap v152: 161 pgs: 33 peering, 14 stale+active+clean, 114 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T04:04:13.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:13 smithi153 ceph-mon[67960]: Health check failed: Reduced data availability: 6 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-03-12T04:04:13.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:13 smithi153 ceph-mon[67960]: osdmap e135: 8 total, 7 up, 8 in 2024-03-12T04:04:13.942 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:13 smithi153 podman[77771]: 2024-03-12 04:04:13.733852969 +0000 UTC m=+0.612884850 container remove bb21ea95d88a393c36948d5b07498a0a5cb9b350cc49567b12bf75ce7b53dc7a (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-12T04:04:13.942 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:13 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-12T04:04:13.942 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:13 smithi153 systemd[1]: Stopped Ceph osd.4 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:04:13.943 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:13 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.4.service: Consumed 16.650s CPU time. 2024-03-12T04:04:14.261 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:13 smithi153 systemd[1]: Starting Ceph osd.4 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:04:14.262 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 podman[77877]: 2024-03-12 04:04:14.159800889 +0000 UTC m=+0.100653207 container create 2f89c65923eaaeb932708bf26422049be4265f8e8f64360cf13a56f05ad551fa (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T04:04:14.262 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 podman[77877]: 2024-03-12 04:04:14.076852974 +0000 UTC m=+0.017705304 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:04:14.692 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 podman[77877]: 2024-03-12 04:04:14.354110864 +0000 UTC m=+0.294963194 container init 2f89c65923eaaeb932708bf26422049be4265f8e8f64360cf13a56f05ad551fa (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-12T04:04:14.692 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 podman[77877]: 2024-03-12 04:04:14.357547081 +0000 UTC m=+0.298399393 container start 2f89c65923eaaeb932708bf26422049be4265f8e8f64360cf13a56f05ad551fa (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e) 2024-03-12T04:04:14.693 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 podman[77877]: 2024-03-12 04:04:14.445483048 +0000 UTC m=+0.386335380 container attach 2f89c65923eaaeb932708bf26422049be4265f8e8f64360cf13a56f05ad551fa (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate, GIT_CLEAN=True, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate[77891]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate[77891]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 bash[77877]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 bash[77877]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 bash[77877]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 bash[77877]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 bash[77877]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 bash[77877]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 bash[77877]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-12T04:04:15.126 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate[77891]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-12T04:04:15.127 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate[77891]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-12T04:04:15.127 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate[77891]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-12T04:04:15.127 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate[77891]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-12T04:04:15.127 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate[77891]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-12T04:04:15.127 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:14 smithi153 podman[77877]: 2024-03-12 04:04:14.819674363 +0000 UTC m=+0.760526668 container died 2f89c65923eaaeb932708bf26422049be4265f8e8f64360cf13a56f05ad551fa (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T04:04:15.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:15 smithi153 ceph-mon[67960]: pgmap v154: 161 pgs: 33 peering, 14 stale+active+clean, 114 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:04:15.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:15 smithi098 ceph-mon[86021]: pgmap v154: 161 pgs: 33 peering, 14 stale+active+clean, 114 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:04:15.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:15 smithi098 ceph-mon[88407]: pgmap v154: 161 pgs: 33 peering, 14 stale+active+clean, 114 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:04:15.757 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 podman[78008]: 2024-03-12 04:04:15.449327406 +0000 UTC m=+0.624173681 container remove 2f89c65923eaaeb932708bf26422049be4265f8e8f64360cf13a56f05ad551fa (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-activate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-12T04:04:15.757 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 podman[78076]: 2024-03-12 04:04:15.567177334 +0000 UTC m=+0.018796820 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:04:15.757 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 podman[78076]: 2024-03-12 04:04:15.667006373 +0000 UTC m=+0.118625840 container create e913f3621dd5181715ac8bbc996baf20cbf0561f8edfc970d901b112247e174c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-12T04:04:16.165 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 podman[78076]: 2024-03-12 04:04:15.869439128 +0000 UTC m=+0.321058622 container init e913f3621dd5181715ac8bbc996baf20cbf0561f8edfc970d901b112247e174c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, ceph=True, org.label-schema.build-date=20240307) 2024-03-12T04:04:16.165 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 podman[78076]: 2024-03-12 04:04:15.873741112 +0000 UTC m=+0.325360589 container start e913f3621dd5181715ac8bbc996baf20cbf0561f8edfc970d901b112247e174c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e) 2024-03-12T04:04:16.165 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 ceph-osd[78096]: -- 172.21.15.153:0/1018604529 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dc856b1680 con 0x55dc8567ac00 2024-03-12T04:04:16.165 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 bash[78076]: e913f3621dd5181715ac8bbc996baf20cbf0561f8edfc970d901b112247e174c 2024-03-12T04:04:16.165 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:15 smithi153 systemd[1]: Started Ceph osd.4 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:04:16.692 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:16 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:04:16.413+0000 7f784e0897c0 -1 Falling back to public interface 2024-03-12T04:04:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:17 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:17 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:17.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:17 smithi153 ceph-mon[67960]: pgmap v155: 161 pgs: 14 active+undersized+degraded, 33 peering, 22 active+undersized, 92 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 61/723 objects degraded (8.437%) 2024-03-12T04:04:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:17 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:17 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:17.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:17 smithi098 ceph-mon[86021]: pgmap v155: 161 pgs: 14 active+undersized+degraded, 33 peering, 22 active+undersized, 92 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 61/723 objects degraded (8.437%) 2024-03-12T04:04:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:17 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:17 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:17.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:17 smithi098 ceph-mon[88407]: pgmap v155: 161 pgs: 14 active+undersized+degraded, 33 peering, 22 active+undersized, 92 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 61/723 objects degraded (8.437%) 2024-03-12T04:04:18.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:18 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 61/723 objects degraded (8.437%), 14 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:18.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:18 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 61/723 objects degraded (8.437%), 14 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:18.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:18 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 61/723 objects degraded (8.437%), 14 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:19.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:19 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:19.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:19 smithi098 ceph-mon[86021]: pgmap v156: 161 pgs: 29 active+undersized+degraded, 40 active+undersized, 92 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-03-12T04:04:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:19 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:19.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:19 smithi098 ceph-mon[88407]: pgmap v156: 161 pgs: 29 active+undersized+degraded, 40 active+undersized, 92 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-03-12T04:04:19.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:19 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:19.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:19 smithi153 ceph-mon[67960]: pgmap v156: 161 pgs: 29 active+undersized+degraded, 40 active+undersized, 92 active+clean; 585 KiB data, 171 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-03-12T04:04:19.442 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:04:19.116+0000 7f784e0897c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-12T04:04:19.442 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:04:19.152+0000 7f784e0897c0 -1 osd.4 133 log_to_monitors true 2024-03-12T04:04:20.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:20 smithi098 ceph-mon[86021]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-03-12T04:04:20.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:20 smithi098 ceph-mon[86021]: from='osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T04:04:20.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:20 smithi098 ceph-mon[86021]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T04:04:20.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:04:20] "GET /metrics HTTP/1.1" 200 38262 "" "Prometheus/2.43.0" 2024-03-12T04:04:20.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:20 smithi098 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-03-12T04:04:20.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:20 smithi098 ceph-mon[88407]: from='osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T04:04:20.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:20 smithi098 ceph-mon[88407]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T04:04:20.442 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:04:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:04:20.058+0000 7f783f4e1700 -1 osd.4 133 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:04:20.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:20 smithi153 ceph-mon[67960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 10 pgs peering) 2024-03-12T04:04:20.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:20 smithi153 ceph-mon[67960]: from='osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T04:04:20.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:20 smithi153 ceph-mon[67960]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-12T04:04:21.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[86021]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[86021]: from='osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[86021]: osdmap e136: 8 total, 7 up, 8 in 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[86021]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[86021]: pgmap v158: 161 pgs: 29 active+undersized+degraded, 40 active+undersized, 92 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[88407]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[88407]: from='osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[88407]: osdmap e136: 8 total, 7 up, 8 in 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[88407]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[88407]: pgmap v158: 161 pgs: 29 active+undersized+degraded, 40 active+undersized, 92 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-03-12T04:04:21.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:21.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:21 smithi153 ceph-mon[67960]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-12T04:04:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:21 smithi153 ceph-mon[67960]: from='osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:21 smithi153 ceph-mon[67960]: osdmap e136: 8 total, 7 up, 8 in 2024-03-12T04:04:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:21 smithi153 ceph-mon[67960]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:21 smithi153 ceph-mon[67960]: pgmap v158: 161 pgs: 29 active+undersized+degraded, 40 active+undersized, 92 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-03-12T04:04:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:22.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:04:22.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[88407]: osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050] boot 2024-03-12T04:04:22.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[88407]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[86021]: osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050] boot 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[86021]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:22.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:22.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:22 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:04:22.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:22 smithi153 ceph-mon[67960]: osd.4 [v2:172.21.15.153:6800/3002368050,v1:172.21.15.153:6801/3002368050] boot 2024-03-12T04:04:22.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:22 smithi153 ceph-mon[67960]: osdmap e137: 8 total, 8 up, 8 in 2024-03-12T04:04:22.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-12T04:04:22.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:22.259 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:23 smithi153 ceph-mon[67960]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T04:04:23.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:23 smithi153 ceph-mon[67960]: pgmap v161: 161 pgs: 18 active+undersized+degraded, 28 active+undersized, 115 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-12T04:04:23.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:23 smithi098 ceph-mon[86021]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T04:04:23.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:23 smithi098 ceph-mon[86021]: pgmap v161: 161 pgs: 18 active+undersized+degraded, 28 active+undersized, 115 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-12T04:04:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:23 smithi098 ceph-mon[88407]: osdmap e138: 8 total, 8 up, 8 in 2024-03-12T04:04:23.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:23 smithi098 ceph-mon[88407]: pgmap v161: 161 pgs: 18 active+undersized+degraded, 28 active+undersized, 115 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-12T04:04:24.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:24 smithi098 ceph-mon[86021]: Health check update: Degraded data redundancy: 71/723 objects degraded (9.820%), 18 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:24.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:24 smithi098 ceph-mon[88407]: Health check update: Degraded data redundancy: 71/723 objects degraded (9.820%), 18 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:24.362 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:24 smithi153 ceph-mon[67960]: Health check update: Degraded data redundancy: 71/723 objects degraded (9.820%), 18 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:25.324 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:24 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:04:24.830+0000 7f79ceb0e700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-12T04:04:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: pgmap v162: 161 pgs: 18 active+undersized+degraded, 28 active+undersized, 115 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-12T04:04:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:25.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:04:25.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:04:25.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.325 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: pgmap v162: 161 pgs: 18 active+undersized+degraded, 28 active+undersized, 115 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.326 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.327 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:25.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: pgmap v162: 161 pgs: 18 active+undersized+degraded, 28 active+undersized, 115 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-03-12T04:04:25.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:25.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:04:25.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:25.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:04:25.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:25.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:26.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:26 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:26.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:26 smithi098 ceph-mon[86021]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-12T04:04:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:26 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:26.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:26 smithi098 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-12T04:04:26.412 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:26 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:26.412 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:26 smithi153 ceph-mon[67960]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-12T04:04:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:27 smithi153 ceph-mon[67960]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T04:04:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:27 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/723 objects degraded (9.820%), 18 pgs degraded) 2024-03-12T04:04:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:27 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:04:27.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[88407]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/723 objects degraded (9.820%), 18 pgs degraded) 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[86021]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 182 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 71/723 objects degraded (9.820%), 18 pgs degraded) 2024-03-12T04:04:27.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:27 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:04:29.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:29 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:29 smithi098 ceph-mon[86021]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:29.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:29 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:29.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:29 smithi098 ceph-mon[88407]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:29 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:29 smithi153 ceph-mon[67960]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:04:30] "GET /metrics HTTP/1.1" 200 38270 "" "Prometheus/2.43.0" 2024-03-12T04:04:31.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:31 smithi098 ceph-mon[88407]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-12T04:04:31.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:31 smithi098 ceph-mon[86021]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-12T04:04:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:31 smithi153 ceph-mon[67960]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-12T04:04:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:32 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:32.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:32 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:32.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:32 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:33 smithi153 ceph-mon[67960]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-12T04:04:33.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:33 smithi098 ceph-mon[86021]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-12T04:04:33.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:33 smithi098 ceph-mon[88407]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-12T04:04:34.027 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:04:34.674 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:04:34.674 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (6m) 48s ago 28m 27.6M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:04:34.674 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (6m) 14s ago 28m 87.8M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:04:34.674 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (6m) 48s ago 27m 62.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:04:34.674 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (7m) 14s ago 33m 452M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:04:34.674 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (17m) 48s ago 34m 508M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (4m) 48s ago 35m 50.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (5m) 14s ago 33m 44.6M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (4m) 48s ago 33m 42.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (7m) 48s ago 29m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (7m) 14s ago 29m 15.3M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (3m) 48s ago 32m 44.9M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (2m) 48s ago 32m 44.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (107s) 48s ago 32m 49.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (53s) 48s ago 31m 11.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (19s) 14s ago 31m 12.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (31m) 14s ago 31m 59.2M 3248M 17.2.0 e1d6a67b021e 7d881042e785 2024-03-12T04:04:34.675 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (30m) 14s ago 30m 56.4M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:04:34.676 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (30m) 14s ago 30m 59.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:04:34.676 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (7m) 14s ago 29m 76.6M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:04:34.676 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (27m) 48s ago 27m 78.4M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:04:34.676 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (27m) 14s ago 27m 81.6M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:04:35.066 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:04:35.066 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 5 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:04:35.067 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:04:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-12T04:04:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 10 2024-03-12T04:04:35.068 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:04:35.068 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:04:35.403 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:04:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:04:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:04:35.403 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-12T04:04:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:04:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "5/8 daemons upgraded", 2024-03-12T04:04:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:04:35.404 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:04:35.404 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:04:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:35 smithi153 ceph-mon[67960]: from='client.64411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:35 smithi153 ceph-mon[67960]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:35 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/3089226331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:35.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:35 smithi098 ceph-mon[86021]: from='client.64411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:35.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:35 smithi098 ceph-mon[86021]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:35.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:35 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/3089226331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:35.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:35 smithi098 ceph-mon[88407]: from='client.64411 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:35.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:35 smithi098 ceph-mon[88407]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:35.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:35 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/3089226331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:36 smithi153 ceph-mon[67960]: from='client.64417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:36 smithi153 ceph-mon[67960]: from='client.64423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:36.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:36 smithi098 ceph-mon[88407]: from='client.64417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:36.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:36 smithi098 ceph-mon[88407]: from='client.64423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:36.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:36 smithi098 ceph-mon[86021]: from='client.64417 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:36.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:36 smithi098 ceph-mon[86021]: from='client.64423 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:37 smithi153 ceph-mon[67960]: from='client.54501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:37 smithi153 ceph-mon[67960]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:37.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:37 smithi098 ceph-mon[86021]: from='client.54501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:37.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:37 smithi098 ceph-mon[86021]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:37.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:37 smithi098 ceph-mon[88407]: from='client.54501 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:04:37.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:37 smithi098 ceph-mon[88407]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:39 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:39 smithi153 ceph-mon[67960]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:39.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:39 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:39.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:39 smithi098 ceph-mon[86021]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:39.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:39 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:39 smithi098 ceph-mon[88407]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:40.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:04:40] "GET /metrics HTTP/1.1" 200 38270 "" "Prometheus/2.43.0" 2024-03-12T04:04:40.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:40.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:40.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[88407]: Upgrade: osd.5 is safe to restart 2024-03-12T04:04:40.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[88407]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:40.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[86021]: Upgrade: osd.5 is safe to restart 2024-03-12T04:04:40.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:40 smithi098 ceph-mon[86021]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:40 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:40 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-12T04:04:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:40 smithi153 ceph-mon[67960]: Upgrade: osd.5 is safe to restart 2024-03-12T04:04:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:40 smithi153 ceph-mon[67960]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:42.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:42.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:42.539 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:42.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T04:04:42.540 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T04:04:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-12T04:04:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:43.554 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:43 smithi153 ceph-mon[67960]: Upgrade: Updating osd.5 2024-03-12T04:04:43.554 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:43 smithi153 ceph-mon[67960]: Deploying daemon osd.5 on smithi153 2024-03-12T04:04:43.554 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:43 smithi153 ceph-mon[67960]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:43.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:43 smithi098 ceph-mon[88407]: Upgrade: Updating osd.5 2024-03-12T04:04:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:43 smithi098 ceph-mon[88407]: Deploying daemon osd.5 on smithi153 2024-03-12T04:04:43.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:43 smithi098 ceph-mon[88407]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:43 smithi098 ceph-mon[86021]: Upgrade: Updating osd.5 2024-03-12T04:04:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:43 smithi098 ceph-mon[86021]: Deploying daemon osd.5 on smithi153 2024-03-12T04:04:43.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:43 smithi098 ceph-mon[86021]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:04:44.769 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:44 smithi153 systemd[1]: Stopping Ceph osd.5 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:04:45.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[31977]: 2024-03-12T04:04:44.765+0000 7fa73e91b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:04:45.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[31977]: 2024-03-12T04:04:44.765+0000 7fa73e91b700 -1 osd.5 138 *** Got signal Terminated *** 2024-03-12T04:04:45.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[31977]: 2024-03-12T04:04:44.765+0000 7fa73e91b700 -1 osd.5 138 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:04:45.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:45 smithi098 ceph-mon[86021]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:45.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:45 smithi098 ceph-mon[86021]: osd.5 marked itself down and dead 2024-03-12T04:04:45.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:45 smithi098 ceph-mon[88407]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:45.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:45 smithi098 ceph-mon[88407]: osd.5 marked itself down and dead 2024-03-12T04:04:45.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:45 smithi153 ceph-mon[67960]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:04:45.590 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:45 smithi153 ceph-mon[67960]: osd.5 marked itself down and dead 2024-03-12T04:04:45.851 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:45 smithi153 podman[82167]: 2024-03-12 04:04:45.589016923 +0000 UTC m=+0.919570796 container died 7d881042e785a538fcbf37973f95fba29d932812a106e8758b5f7b248409bd92 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, release=754, name=centos-stream, distribution-scope=public, GIT_BRANCH=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-12T04:04:46.152 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:45 smithi153 podman[82167]: 2024-03-12 04:04:45.924189502 +0000 UTC m=+1.254743368 container cleanup 7d881042e785a538fcbf37973f95fba29d932812a106e8758b5f7b248409bd92 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, name=centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, RELEASE=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc.) 2024-03-12T04:04:46.152 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:45 smithi153 bash[82167]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5 2024-03-12T04:04:46.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:46 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:04:46.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:46 smithi153 ceph-mon[67960]: osdmap e139: 8 total, 7 up, 8 in 2024-03-12T04:04:46.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:46 smithi153 ceph-mon[67960]: osdmap e140: 8 total, 7 up, 8 in 2024-03-12T04:04:46.442 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:46 smithi153 podman[82194]: 2024-03-12 04:04:46.150641314 +0000 UTC m=+0.562097292 container remove 7d881042e785a538fcbf37973f95fba29d932812a106e8758b5f7b248409bd92 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-type=git, GIT_CLEAN=True, ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-03-12T04:04:46.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:46 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:04:46.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:46 smithi098 ceph-mon[86021]: osdmap e139: 8 total, 7 up, 8 in 2024-03-12T04:04:46.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:46 smithi098 ceph-mon[86021]: osdmap e140: 8 total, 7 up, 8 in 2024-03-12T04:04:46.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:46 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:04:46.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:46 smithi098 ceph-mon[88407]: osdmap e139: 8 total, 7 up, 8 in 2024-03-12T04:04:46.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:46 smithi098 ceph-mon[88407]: osdmap e140: 8 total, 7 up, 8 in 2024-03-12T04:04:46.856 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:46 smithi153 podman[82290]: 2024-03-12 04:04:46.505927028 +0000 UTC m=+0.018719495 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:04:46.856 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:46 smithi153 podman[82290]: 2024-03-12 04:04:46.626562938 +0000 UTC m=+0.139355396 container create d55feb0e89688d39b15b57249a4c5b4c56b4386d16542e2ebe23dd5108621799 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-12T04:04:46.856 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:46 smithi153 podman[82290]: 2024-03-12 04:04:46.852050749 +0000 UTC m=+0.364843222 container init d55feb0e89688d39b15b57249a4c5b4c56b4386d16542e2ebe23dd5108621799 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e) 2024-03-12T04:04:46.856 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:46 smithi153 podman[82290]: 2024-03-12 04:04:46.855055664 +0000 UTC m=+0.367848123 container start d55feb0e89688d39b15b57249a4c5b4c56b4386d16542e2ebe23dd5108621799 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-12T04:04:47.155 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:46 smithi153 podman[82290]: 2024-03-12 04:04:46.943872239 +0000 UTC m=+0.456664712 container attach d55feb0e89688d39b15b57249a4c5b4c56b4386d16542e2ebe23dd5108621799 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:04:47.440 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:47 smithi153 ceph-mon[67960]: pgmap v174: 161 pgs: 34 peering, 10 stale+active+clean, 117 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:04:47.441 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:47 smithi153 ceph-mon[67960]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-12T04:04:47.441 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:47 smithi153 podman[82290]: 2024-03-12 04:04:47.153997869 +0000 UTC m=+0.666790320 container died d55feb0e89688d39b15b57249a4c5b4c56b4386d16542e2ebe23dd5108621799 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=) 2024-03-12T04:04:47.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:47 smithi098 ceph-mon[86021]: pgmap v174: 161 pgs: 34 peering, 10 stale+active+clean, 117 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:04:47.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:47 smithi098 ceph-mon[86021]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-12T04:04:47.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:47 smithi098 ceph-mon[88407]: pgmap v174: 161 pgs: 34 peering, 10 stale+active+clean, 117 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail 2024-03-12T04:04:47.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:47 smithi098 ceph-mon[88407]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-12T04:04:47.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:47 smithi153 podman[82316]: 2024-03-12 04:04:47.664344336 +0000 UTC m=+0.501497389 container remove d55feb0e89688d39b15b57249a4c5b4c56b4386d16542e2ebe23dd5108621799 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:04:47.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:47 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-12T04:04:47.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:47 smithi153 systemd[1]: Stopped Ceph osd.5 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:04:47.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:47 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.5.service: Consumed 16.153s CPU time. 2024-03-12T04:04:48.104 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:47 smithi153 systemd[1]: Starting Ceph osd.5 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:04:48.361 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 19/723 objects degraded (2.628%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:48.361 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 podman[82423]: 2024-03-12 04:04:48.004601085 +0000 UTC m=+0.017665136 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:04:48.362 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 podman[82423]: 2024-03-12 04:04:48.123727265 +0000 UTC m=+0.136791351 container create 3417cdbe9d7b7abbb5c4a32f54a705ab65497e2ef25fff100aa6eaa2130fe50b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:04:48.362 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 podman[82423]: 2024-03-12 04:04:48.360164679 +0000 UTC m=+0.373228746 container init 3417cdbe9d7b7abbb5c4a32f54a705ab65497e2ef25fff100aa6eaa2130fe50b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-12T04:04:48.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:48 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 19/723 objects degraded (2.628%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:48.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:48 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 19/723 objects degraded (2.628%), 6 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:48.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 podman[82423]: 2024-03-12 04:04:48.363517019 +0000 UTC m=+0.376581073 container start 3417cdbe9d7b7abbb5c4a32f54a705ab65497e2ef25fff100aa6eaa2130fe50b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T04:04:48.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 podman[82423]: 2024-03-12 04:04:48.434546681 +0000 UTC m=+0.447610737 container attach 3417cdbe9d7b7abbb5c4a32f54a705ab65497e2ef25fff100aa6eaa2130fe50b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate[82439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate[82439]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 bash[82423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 bash[82423]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 bash[82423]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 bash[82423]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 bash[82423]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 bash[82423]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 bash[82423]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate[82439]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate[82439]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-12T04:04:49.131 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate[82439]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-12T04:04:49.132 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate[82439]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-12T04:04:49.132 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate[82439]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-12T04:04:49.132 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:48 smithi153 podman[82423]: 2024-03-12 04:04:48.830961784 +0000 UTC m=+0.844025839 container died 3417cdbe9d7b7abbb5c4a32f54a705ab65497e2ef25fff100aa6eaa2130fe50b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:04:49.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:49 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:49.406 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:49 smithi153 ceph-mon[67960]: pgmap v176: 161 pgs: 6 active+undersized+degraded, 34 peering, 4 stale+active+clean, 9 active+undersized, 108 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 19/723 objects degraded (2.628%) 2024-03-12T04:04:49.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:49 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:49.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:49 smithi098 ceph-mon[86021]: pgmap v176: 161 pgs: 6 active+undersized+degraded, 34 peering, 4 stale+active+clean, 9 active+undersized, 108 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 19/723 objects degraded (2.628%) 2024-03-12T04:04:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:49 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:49 smithi098 ceph-mon[88407]: pgmap v176: 161 pgs: 6 active+undersized+degraded, 34 peering, 4 stale+active+clean, 9 active+undersized, 108 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 19/723 objects degraded (2.628%) 2024-03-12T04:04:49.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 podman[82560]: 2024-03-12 04:04:49.404943394 +0000 UTC m=+0.568665536 container remove 3417cdbe9d7b7abbb5c4a32f54a705ab65497e2ef25fff100aa6eaa2130fe50b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T04:04:49.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 podman[82626]: 2024-03-12 04:04:49.521836357 +0000 UTC m=+0.018587601 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:04:49.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 podman[82626]: 2024-03-12 04:04:49.622572428 +0000 UTC m=+0.119323664 container create 42f3b944dd81fc9f10d04d5d3a761074abbe01fe067deee60c9793858b6cac96 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:04:49.994 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 podman[82626]: 2024-03-12 04:04:49.891599834 +0000 UTC m=+0.388351070 container init 42f3b944dd81fc9f10d04d5d3a761074abbe01fe067deee60c9793858b6cac96 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-12T04:04:49.995 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 podman[82626]: 2024-03-12 04:04:49.895365886 +0000 UTC m=+0.392117123 container start 42f3b944dd81fc9f10d04d5d3a761074abbe01fe067deee60c9793858b6cac96 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-2b0101e) 2024-03-12T04:04:49.995 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 ceph-osd[82645]: -- 172.21.15.153:0/241776828 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55daa8e53680 con 0x55daa8e1d000 2024-03-12T04:04:49.995 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 bash[82626]: 42f3b944dd81fc9f10d04d5d3a761074abbe01fe067deee60c9793858b6cac96 2024-03-12T04:04:50.442 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:49 smithi153 systemd[1]: Started Ceph osd.5 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:04:50.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:04:50] "GET /metrics HTTP/1.1" 200 38270 "" "Prometheus/2.43.0" 2024-03-12T04:04:50.839 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:50 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:04:50.442+0000 7f3e2a1587c0 -1 Falling back to public interface 2024-03-12T04:04:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:51 smithi153 ceph-mon[67960]: pgmap v177: 161 pgs: 10 active+undersized+degraded, 34 peering, 15 active+undersized, 102 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 35/723 objects degraded (4.841%) 2024-03-12T04:04:51.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:51.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:51.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:51 smithi098 ceph-mon[86021]: pgmap v177: 161 pgs: 10 active+undersized+degraded, 34 peering, 15 active+undersized, 102 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 35/723 objects degraded (4.841%) 2024-03-12T04:04:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:51.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:51 smithi098 ceph-mon[88407]: pgmap v177: 161 pgs: 10 active+undersized+degraded, 34 peering, 15 active+undersized, 102 active+clean; 585 KiB data, 186 MiB used, 715 GiB / 715 GiB avail; 35/723 objects degraded (4.841%) 2024-03-12T04:04:52.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:52 smithi098 ceph-mon[86021]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-03-12T04:04:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:52 smithi098 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-03-12T04:04:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:52 smithi153 ceph-mon[67960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-03-12T04:04:53.442 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:04:53.130+0000 7f3e2a1587c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-12T04:04:53.624 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:04:53.162+0000 7f3e2a1587c0 -1 osd.5 138 log_to_monitors true 2024-03-12T04:04:53.624 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:04:53 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:04:53.352+0000 7f3e1b5b0700 -1 osd.5 138 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:04:53.624 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:53 smithi153 ceph-mon[67960]: pgmap v178: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 79/723 objects degraded (10.927%) 2024-03-12T04:04:53.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:53 smithi153 ceph-mon[67960]: from='osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T04:04:53.625 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:53 smithi153 ceph-mon[67960]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T04:04:53.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:53 smithi098 ceph-mon[86021]: pgmap v178: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 79/723 objects degraded (10.927%) 2024-03-12T04:04:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:53 smithi098 ceph-mon[86021]: from='osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T04:04:53.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:53 smithi098 ceph-mon[86021]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T04:04:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:53 smithi098 ceph-mon[88407]: pgmap v178: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 195 MiB used, 715 GiB / 715 GiB avail; 79/723 objects degraded (10.927%) 2024-03-12T04:04:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:53 smithi098 ceph-mon[88407]: from='osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T04:04:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:53 smithi098 ceph-mon[88407]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-12T04:04:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:54 smithi153 ceph-mon[67960]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T04:04:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:54 smithi153 ceph-mon[67960]: osdmap e141: 8 total, 7 up, 8 in 2024-03-12T04:04:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:54 smithi153 ceph-mon[67960]: from='osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:54.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:54 smithi153 ceph-mon[67960]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:54.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[86021]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T04:04:54.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[86021]: osdmap e141: 8 total, 7 up, 8 in 2024-03-12T04:04:54.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[86021]: from='osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:54.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[86021]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[88407]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-12T04:04:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[88407]: osdmap e141: 8 total, 7 up, 8 in 2024-03-12T04:04:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[88407]: from='osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:54.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:54 smithi098 ceph-mon[88407]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:04:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: pgmap v180: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 79/723 objects degraded (10.927%) 2024-03-12T04:04:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:04:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844] boot 2024-03-12T04:04:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T04:04:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T04:04:55.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:55.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:55.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:55 smithi153 ceph-mon[67960]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T04:04:55.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: pgmap v180: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 79/723 objects degraded (10.927%) 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844] boot 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[88407]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: pgmap v180: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 79/723 objects degraded (10.927%) 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: osd.5 [v2:172.21.15.153:6808/2619468844,v1:172.21.15.153:6809/2619468844] boot 2024-03-12T04:04:55.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: osdmap e142: 8 total, 8 up, 8 in 2024-03-12T04:04:55.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-12T04:04:55.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:55.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:55.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:55 smithi098 ceph-mon[86021]: osdmap e143: 8 total, 8 up, 8 in 2024-03-12T04:04:56.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[86021]: pgmap v183: 161 pgs: 14 peering, 11 active+undersized+degraded, 20 active+undersized, 116 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 46/723 objects degraded (6.362%) 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[86021]: Health check update: Degraded data redundancy: 79/723 objects degraded (10.927%), 21 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[88407]: pgmap v183: 161 pgs: 14 peering, 11 active+undersized+degraded, 20 active+undersized, 116 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 46/723 objects degraded (6.362%) 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[88407]: Health check update: Degraded data redundancy: 79/723 objects degraded (10.927%), 21 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:56.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:56 smithi153 ceph-mon[67960]: pgmap v183: 161 pgs: 14 peering, 11 active+undersized+degraded, 20 active+undersized, 116 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 46/723 objects degraded (6.362%) 2024-03-12T04:04:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:56 smithi153 ceph-mon[67960]: Health check update: Degraded data redundancy: 79/723 objects degraded (10.927%), 21 pgs degraded (PG_DEGRADED) 2024-03-12T04:04:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:04:56.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.273 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:04:58 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:04:58.922+0000 7f79ceb0e700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: pgmap v184: 161 pgs: 14 peering, 7 active+undersized+degraded, 15 active+undersized, 125 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 34/723 objects degraded (4.703%) 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: pgmap v184: 161 pgs: 14 peering, 7 active+undersized+degraded, 15 active+undersized, 125 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 34/723 objects degraded (4.703%) 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:04:59 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: pgmap v184: 161 pgs: 14 peering, 7 active+undersized+degraded, 15 active+undersized, 125 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 34/723 objects degraded (4.703%) 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:04:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:04:59.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:04:59 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:00.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:00 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:00.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:00 smithi098 ceph-mon[86021]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-12T04:05:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:05:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:05:00] "GET /metrics HTTP/1.1" 200 38131 "" "Prometheus/2.43.0" 2024-03-12T04:05:00.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:00 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:00.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:00 smithi098 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-12T04:05:00.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:00 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:00.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:00 smithi153 ceph-mon[67960]: Upgrade: unsafe to stop osd(s) at this time (10 PGs are or would become offline) 2024-03-12T04:05:01.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:01 smithi098 ceph-mon[88407]: pgmap v185: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-12T04:05:01.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:01 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/723 objects degraded (4.703%), 7 pgs degraded) 2024-03-12T04:05:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:01 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:05:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:01 smithi098 ceph-mon[86021]: pgmap v185: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-12T04:05:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:01 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/723 objects degraded (4.703%), 7 pgs degraded) 2024-03-12T04:05:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:01 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:05:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:01 smithi153 ceph-mon[67960]: pgmap v185: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 op/s 2024-03-12T04:05:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:01 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 34/723 objects degraded (4.703%), 7 pgs degraded) 2024-03-12T04:05:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:01 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:05:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:03 smithi098 ceph-mon[86021]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:05:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:03 smithi098 ceph-mon[88407]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:05:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:03 smithi153 ceph-mon[67960]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:05:05.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:05 smithi098 ceph-mon[86021]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-12T04:05:05.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:05 smithi098 ceph-mon[88407]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-12T04:05:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:05 smithi153 ceph-mon[67960]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 932 B/s rd, 0 op/s 2024-03-12T04:05:05.759 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (7m) 80s ago 29m 27.6M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (7m) 11s ago 28m 89.5M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (6m) 80s ago 28m 62.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (8m) 11s ago 33m 452M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (18m) 80s ago 35m 508M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (5m) 80s ago 35m 50.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (6m) 11s ago 33m 46.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (4m) 80s ago 34m 42.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (8m) 80s ago 30m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:05:06.413 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (8m) 11s ago 29m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (3m) 80s ago 33m 44.9M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (3m) 80s ago 33m 44.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (2m) 80s ago 32m 49.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (85s) 80s ago 32m 11.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (50s) 11s ago 31m 44.5M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (16s) 11s ago 31m 12.5M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (31m) 11s ago 31m 56.8M 3248M 17.2.0 e1d6a67b021e 94e531f2efb5 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (30m) 11s ago 30m 60.2M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (7m) 11s ago 30m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (28m) 80s ago 28m 78.4M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:05:06.414 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (28m) 11s ago 28m 81.8M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-12T04:05:06.806 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 6 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 11 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:05:06.807 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:05:07.132 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "6/8 daemons upgraded", 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:05:07.133 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[88407]: from='client.54513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[88407]: from='client.64453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[88407]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1832262524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[86021]: from='client.54513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[86021]: from='client.64453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:07.383 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[86021]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:05:07.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:07.384 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:07 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1832262524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:07 smithi153 ceph-mon[67960]: from='client.54513 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:07 smithi153 ceph-mon[67960]: from='client.64453 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:07 smithi153 ceph-mon[67960]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-12T04:05:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:07 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1832262524' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:08.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:08 smithi098 ceph-mon[88407]: from='client.64459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:08.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:08 smithi098 ceph-mon[88407]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:08.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:08 smithi098 ceph-mon[86021]: from='client.64459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:08.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:08 smithi098 ceph-mon[86021]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:08 smithi153 ceph-mon[67960]: from='client.64459 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:08 smithi153 ceph-mon[67960]: from='client.64471 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:09.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:09 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:09.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:09 smithi098 ceph-mon[88407]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:09.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:09 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:09.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:09 smithi098 ceph-mon[86021]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:09 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:09 smithi153 ceph-mon[67960]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:10.474 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:05:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:05:10] "GET /metrics HTTP/1.1" 200 38131 "" "Prometheus/2.43.0" 2024-03-12T04:05:11.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:11 smithi098 ceph-mon[86021]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:11.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:11.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:11 smithi098 ceph-mon[88407]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:11.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:11 smithi153 ceph-mon[67960]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:13 smithi098 ceph-mon[86021]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:13 smithi098 ceph-mon[88407]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:13 smithi153 ceph-mon[67960]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:14.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:14 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:14.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:14 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:14.615 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:14 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:15.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:15 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:15.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:15 smithi153 ceph-mon[67960]: Upgrade: osd.6 is safe to restart 2024-03-12T04:05:15.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:15 smithi153 ceph-mon[67960]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:15.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:15 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:15.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:15 smithi098 ceph-mon[86021]: Upgrade: osd.6 is safe to restart 2024-03-12T04:05:15.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:15 smithi098 ceph-mon[86021]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:15.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:15 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-12T04:05:15.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:15 smithi098 ceph-mon[88407]: Upgrade: osd.6 is safe to restart 2024-03-12T04:05:15.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:15 smithi098 ceph-mon[88407]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:16.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:16 smithi153 ceph-mon[67960]: Upgrade: Updating osd.6 2024-03-12T04:05:16.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:16 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:16.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:16 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T04:05:16.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:16 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:16.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:16 smithi153 ceph-mon[67960]: Deploying daemon osd.6 on smithi153 2024-03-12T04:05:16.664 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:16 smithi153 ceph-mon[67960]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:16.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[86021]: Upgrade: Updating osd.6 2024-03-12T04:05:16.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[86021]: Deploying daemon osd.6 on smithi153 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[86021]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[88407]: Upgrade: Updating osd.6 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[88407]: Deploying daemon osd.6 on smithi153 2024-03-12T04:05:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:16 smithi098 ceph-mon[88407]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:18.942 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:18 smithi153 systemd[1]: Stopping Ceph osd.6 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:05:18.943 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[34867]: 2024-03-12T04:05:18.739+0000 7f30bbb24700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:05:18.943 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[34867]: 2024-03-12T04:05:18.739+0000 7f30bbb24700 -1 osd.6 143 *** Got signal Terminated *** 2024-03-12T04:05:18.943 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[34867]: 2024-03-12T04:05:18.739+0000 7f30bbb24700 -1 osd.6 143 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:05:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:19 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:19 smithi098 ceph-mon[86021]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:19.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:19 smithi098 ceph-mon[86021]: osd.6 marked itself down and dead 2024-03-12T04:05:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:19 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:19 smithi098 ceph-mon[88407]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:19 smithi098 ceph-mon[88407]: osd.6 marked itself down and dead 2024-03-12T04:05:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:19 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:19 smithi153 ceph-mon[67960]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:19 smithi153 ceph-mon[67960]: osd.6 marked itself down and dead 2024-03-12T04:05:19.991 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:19 smithi153 podman[87135]: 2024-03-12 04:05:19.731253724 +0000 UTC m=+1.077546296 container died 94e531f2efb5484520c8c50be6fbaee2a49be1527c5579287d988db5188a6b8b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.component=centos-stream-container, release=754, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, ceph=True) 2024-03-12T04:05:20.281 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 podman[87135]: 2024-03-12 04:05:20.089375133 +0000 UTC m=+1.435667703 container cleanup 94e531f2efb5484520c8c50be6fbaee2a49be1527c5579287d988db5188a6b8b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, ceph=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_CLEAN=True, vendor=Red Hat, Inc., io.openshift.expose-services=, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.tags=base centos centos-stream, distribution-scope=public, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-type=git) 2024-03-12T04:05:20.281 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 bash[87135]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6 2024-03-12T04:05:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:05:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:05:20] "GET /metrics HTTP/1.1" 200 38275 "" "Prometheus/2.43.0" 2024-03-12T04:05:20.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:20 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:05:20.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:20 smithi098 ceph-mon[86021]: osdmap e144: 8 total, 7 up, 8 in 2024-03-12T04:05:20.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:20 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:05:20.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:20 smithi098 ceph-mon[88407]: osdmap e144: 8 total, 7 up, 8 in 2024-03-12T04:05:20.692 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 podman[87153]: 2024-03-12 04:05:20.340549382 +0000 UTC m=+0.603279792 container remove 94e531f2efb5484520c8c50be6fbaee2a49be1527c5579287d988db5188a6b8b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, RELEASE=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vcs-type=git, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, ceph=True, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-03-12T04:05:20.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:20 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:05:20.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:20 smithi153 ceph-mon[67960]: osdmap e144: 8 total, 7 up, 8 in 2024-03-12T04:05:20.970 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 podman[87247]: 2024-03-12 04:05:20.602999757 +0000 UTC m=+0.018136305 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:05:20.970 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 podman[87247]: 2024-03-12 04:05:20.716664152 +0000 UTC m=+0.131800695 container create f405b2897b6b05df37bb304f1dc16a9b84b9e7c8f78c8273f900f2e71abaa1f4 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-12T04:05:20.971 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 podman[87247]: 2024-03-12 04:05:20.894261811 +0000 UTC m=+0.309398368 container init f405b2897b6b05df37bb304f1dc16a9b84b9e7c8f78c8273f900f2e71abaa1f4 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307) 2024-03-12T04:05:20.971 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 podman[87247]: 2024-03-12 04:05:20.897418064 +0000 UTC m=+0.312554608 container start f405b2897b6b05df37bb304f1dc16a9b84b9e7c8f78c8273f900f2e71abaa1f4 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T04:05:20.971 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:20 smithi153 podman[87247]: 2024-03-12 04:05:20.969057933 +0000 UTC m=+0.384194486 container attach f405b2897b6b05df37bb304f1dc16a9b84b9e7c8f78c8273f900f2e71abaa1f4 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-12T04:05:21.285 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:21 smithi153 podman[87247]: 2024-03-12 04:05:21.192490355 +0000 UTC m=+0.607626907 container died f405b2897b6b05df37bb304f1dc16a9b84b9e7c8f78c8273f900f2e71abaa1f4 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-12T04:05:21.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:21 smithi098 ceph-mon[86021]: pgmap v196: 161 pgs: 15 peering, 7 stale+active+clean, 139 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T04:05:21.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:21 smithi098 ceph-mon[86021]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:21.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:21 smithi098 ceph-mon[86021]: osdmap e145: 8 total, 7 up, 8 in 2024-03-12T04:05:21.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:21 smithi098 ceph-mon[88407]: pgmap v196: 161 pgs: 15 peering, 7 stale+active+clean, 139 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T04:05:21.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:21 smithi098 ceph-mon[88407]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:21.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:21 smithi098 ceph-mon[88407]: osdmap e145: 8 total, 7 up, 8 in 2024-03-12T04:05:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:21 smithi153 ceph-mon[67960]: pgmap v196: 161 pgs: 15 peering, 7 stale+active+clean, 139 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T04:05:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:21 smithi153 ceph-mon[67960]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:21 smithi153 ceph-mon[67960]: osdmap e145: 8 total, 7 up, 8 in 2024-03-12T04:05:22.180 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:21 smithi153 podman[87273]: 2024-03-12 04:05:21.730743967 +0000 UTC m=+0.533130654 container remove f405b2897b6b05df37bb304f1dc16a9b84b9e7c8f78c8273f900f2e71abaa1f4 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-12T04:05:22.180 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:21 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-12T04:05:22.180 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:21 smithi153 systemd[1]: Stopped Ceph osd.6 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:05:22.180 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:21 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6.service: Consumed 14.150s CPU time. 2024-03-12T04:05:22.180 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:21 smithi153 systemd[1]: Starting Ceph osd.6 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:05:22.431 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 34/723 objects degraded (4.703%), 10 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:22.431 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 podman[87378]: 2024-03-12 04:05:22.081163324 +0000 UTC m=+0.018077543 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:05:22.431 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 podman[87378]: 2024-03-12 04:05:22.198435309 +0000 UTC m=+0.135349522 container create 36e1eff127fbf68aa75d81a7f56689436a38916629de8feb385a6674490b5e0c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate, GIT_CLEAN=True, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:05:22.431 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 podman[87378]: 2024-03-12 04:05:22.426312279 +0000 UTC m=+0.363226503 container init 36e1eff127fbf68aa75d81a7f56689436a38916629de8feb385a6674490b5e0c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T04:05:22.692 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 podman[87378]: 2024-03-12 04:05:22.429753512 +0000 UTC m=+0.366667726 container start 36e1eff127fbf68aa75d81a7f56689436a38916629de8feb385a6674490b5e0c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-12T04:05:22.692 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 podman[87378]: 2024-03-12 04:05:22.526352199 +0000 UTC m=+0.463266433 container attach 36e1eff127fbf68aa75d81a7f56689436a38916629de8feb385a6674490b5e0c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T04:05:22.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:22 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 34/723 objects degraded (4.703%), 10 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:22.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:22 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 34/723 objects degraded (4.703%), 10 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:23.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate[87393]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-12T04:05:23.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate[87393]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-12T04:05:23.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 bash[87378]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 bash[87378]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 bash[87378]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 bash[87378]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 bash[87378]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 bash[87378]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 bash[87378]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate[87393]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate[87393]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate[87393]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate[87393]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate[87393]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-12T04:05:23.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:22 smithi153 podman[87378]: 2024-03-12 04:05:22.895755668 +0000 UTC m=+0.832669887 container died 36e1eff127fbf68aa75d81a7f56689436a38916629de8feb385a6674490b5e0c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T04:05:23.598 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:23 smithi153 ceph-mon[67960]: pgmap v198: 161 pgs: 10 active+undersized+degraded, 19 peering, 2 stale+active+clean, 14 active+undersized, 116 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 34/723 objects degraded (4.703%) 2024-03-12T04:05:23.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:23 smithi098 ceph-mon[86021]: pgmap v198: 161 pgs: 10 active+undersized+degraded, 19 peering, 2 stale+active+clean, 14 active+undersized, 116 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 34/723 objects degraded (4.703%) 2024-03-12T04:05:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:23 smithi098 ceph-mon[88407]: pgmap v198: 161 pgs: 10 active+undersized+degraded, 19 peering, 2 stale+active+clean, 14 active+undersized, 116 active+clean; 585 KiB data, 210 MiB used, 715 GiB / 715 GiB avail; 34/723 objects degraded (4.703%) 2024-03-12T04:05:23.927 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:23 smithi153 podman[87520]: 2024-03-12 04:05:23.596774317 +0000 UTC m=+0.693338544 container remove 36e1eff127fbf68aa75d81a7f56689436a38916629de8feb385a6674490b5e0c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-activate, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-12T04:05:23.927 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:23 smithi153 podman[87583]: 2024-03-12 04:05:23.713368433 +0000 UTC m=+0.018209599 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:05:23.927 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:23 smithi153 podman[87583]: 2024-03-12 04:05:23.822619483 +0000 UTC m=+0.127460678 container create 23a1ba5d3665f5c70a40546fbfaa115d491682b5c78059c9ab4437c2de40fc54 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T04:05:24.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:24 smithi153 podman[87583]: 2024-03-12 04:05:24.016549321 +0000 UTC m=+0.321390488 container init 23a1ba5d3665f5c70a40546fbfaa115d491682b5c78059c9ab4437c2de40fc54 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-12T04:05:24.509 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:24 smithi153 podman[87583]: 2024-03-12 04:05:24.019652776 +0000 UTC m=+0.324493941 container start 23a1ba5d3665f5c70a40546fbfaa115d491682b5c78059c9ab4437c2de40fc54 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-12T04:05:24.509 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:24 smithi153 ceph-osd[87603]: -- 172.21.15.153:0/233046567 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bb47129680 con 0x55bb470f3000 2024-03-12T04:05:24.509 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:24 smithi153 bash[87583]: 23a1ba5d3665f5c70a40546fbfaa115d491682b5c78059c9ab4437c2de40fc54 2024-03-12T04:05:24.509 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:24 smithi153 systemd[1]: Started Ceph osd.6 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:05:24.914 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:05:24.564+0000 7f48c3f387c0 -1 Falling back to public interface 2024-03-12T04:05:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:25.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:25 smithi153 ceph-mon[67960]: pgmap v199: 161 pgs: 13 active+undersized+degraded, 19 peering, 21 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 46/723 objects degraded (6.362%) 2024-03-12T04:05:25.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:25.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:25.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:25 smithi098 ceph-mon[86021]: pgmap v199: 161 pgs: 13 active+undersized+degraded, 19 peering, 21 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 46/723 objects degraded (6.362%) 2024-03-12T04:05:25.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:25.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:25.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:25 smithi098 ceph-mon[88407]: pgmap v199: 161 pgs: 13 active+undersized+degraded, 19 peering, 21 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 46/723 objects degraded (6.362%) 2024-03-12T04:05:27.442 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:05:27.283+0000 7f48c3f387c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-12T04:05:27.442 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:05:27.320+0000 7f48c3f387c0 -1 osd.6 143 log_to_monitors true 2024-03-12T04:05:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:27 smithi153 ceph-mon[67960]: pgmap v200: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 85/723 objects degraded (11.757%) 2024-03-12T04:05:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:27 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:27 smithi153 ceph-mon[67960]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:27.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:27 smithi153 ceph-mon[67960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-03-12T04:05:27.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[86021]: pgmap v200: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 85/723 objects degraded (11.757%) 2024-03-12T04:05:27.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[86021]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[86021]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[88407]: pgmap v200: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 85/723 objects degraded (11.757%) 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[88407]: Health check update: Reduced data availability: 5 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:27.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:27 smithi098 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 7 pgs peering) 2024-03-12T04:05:28.551 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:28 smithi153 ceph-mon[67960]: from='osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T04:05:28.552 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:28 smithi153 ceph-mon[67960]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T04:05:28.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:28 smithi098 ceph-mon[86021]: from='osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T04:05:28.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:28 smithi098 ceph-mon[86021]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T04:05:28.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:28 smithi098 ceph-mon[88407]: from='osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T04:05:28.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:28 smithi098 ceph-mon[88407]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-12T04:05:28.942 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:05:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:05:28.736+0000 7f48b1388700 -1 osd.6 143 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: pgmap v201: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 85/723 objects degraded (11.757%) 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: osdmap e146: 8 total, 7 up, 8 in 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: from='osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[86021]: from='osd.6 ' entity='osd.6' 2024-03-12T04:05:29.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: pgmap v201: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 85/723 objects degraded (11.757%) 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: osdmap e146: 8 total, 7 up, 8 in 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: from='osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:29.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:29 smithi098 ceph-mon[88407]: from='osd.6 ' entity='osd.6' 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: pgmap v201: 161 pgs: 19 active+undersized+degraded, 34 active+undersized, 108 active+clean; 585 KiB data, 211 MiB used, 715 GiB / 715 GiB avail; 85/723 objects degraded (11.757%) 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: osdmap e146: 8 total, 7 up, 8 in 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: from='osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:29 smithi153 ceph-mon[67960]: from='osd.6 ' entity='osd.6' 2024-03-12T04:05:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:05:30] "GET /metrics HTTP/1.1" 200 38281 "" "Prometheus/2.43.0" 2024-03-12T04:05:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:05:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[86021]: osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699] boot 2024-03-12T04:05:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[86021]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T04:05:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T04:05:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[86021]: pgmap v204: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 68/723 objects degraded (9.405%) 2024-03-12T04:05:30.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:30.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:30.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:05:30.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[88407]: osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699] boot 2024-03-12T04:05:30.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[88407]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T04:05:30.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T04:05:30.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:30 smithi098 ceph-mon[88407]: pgmap v204: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 68/723 objects degraded (9.405%) 2024-03-12T04:05:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:30 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:05:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:30 smithi153 ceph-mon[67960]: osd.6 [v2:172.21.15.153:6816/3315786699,v1:172.21.15.153:6817/3315786699] boot 2024-03-12T04:05:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:30 smithi153 ceph-mon[67960]: osdmap e147: 8 total, 8 up, 8 in 2024-03-12T04:05:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-12T04:05:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:30 smithi153 ceph-mon[67960]: pgmap v204: 161 pgs: 9 peering, 16 active+undersized+degraded, 28 active+undersized, 108 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 68/723 objects degraded (9.405%) 2024-03-12T04:05:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:31 smithi153 ceph-mon[67960]: Health check update: Degraded data redundancy: 68/723 objects degraded (9.405%), 16 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:31 smithi153 ceph-mon[67960]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T04:05:31.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:31 smithi098 ceph-mon[86021]: Health check update: Degraded data redundancy: 68/723 objects degraded (9.405%), 16 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:31.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:31 smithi098 ceph-mon[86021]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T04:05:31.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:31 smithi098 ceph-mon[88407]: Health check update: Degraded data redundancy: 68/723 objects degraded (9.405%), 16 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:31.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:31 smithi098 ceph-mon[88407]: osdmap e148: 8 total, 8 up, 8 in 2024-03-12T04:05:32.618 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:32 smithi153 ceph-mon[67960]: pgmap v206: 161 pgs: 10 activating, 13 peering, 7 active+undersized+degraded, 14 active+undersized, 117 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/723 objects degraded (3.596%) 2024-03-12T04:05:32.818 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:32 smithi098 ceph-mon[86021]: pgmap v206: 161 pgs: 10 activating, 13 peering, 7 active+undersized+degraded, 14 active+undersized, 117 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/723 objects degraded (3.596%) 2024-03-12T04:05:32.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:32 smithi098 ceph-mon[88407]: pgmap v206: 161 pgs: 10 activating, 13 peering, 7 active+undersized+degraded, 14 active+undersized, 117 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/723 objects degraded (3.596%) 2024-03-12T04:05:33.073 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:05:32 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:05:32.816+0000 7f79ceb0e700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-12T04:05:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:33.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:33 smithi153 ceph-mon[67960]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-12T04:05:34.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:34.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[88407]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:34.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:34.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:34.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:33 smithi098 ceph-mon[86021]: Upgrade: unsafe to stop osd(s) at this time (11 PGs are or would become offline) 2024-03-12T04:05:34.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:34 smithi153 ceph-mon[67960]: pgmap v207: 161 pgs: 10 activating, 13 peering, 138 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:05:35.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:34 smithi098 ceph-mon[86021]: pgmap v207: 161 pgs: 10 activating, 13 peering, 138 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:05:35.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:34 smithi098 ceph-mon[88407]: pgmap v207: 161 pgs: 10 activating, 13 peering, 138 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:05:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:35 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/723 objects degraded (3.596%), 7 pgs degraded) 2024-03-12T04:05:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:35 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:05:36.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:35 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/723 objects degraded (3.596%), 7 pgs degraded) 2024-03-12T04:05:36.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:35 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:05:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:35 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/723 objects degraded (3.596%), 7 pgs degraded) 2024-03-12T04:05:36.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:35 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:05:36.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:36 smithi153 ceph-mon[67960]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T04:05:37.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:36 smithi098 ceph-mon[86021]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T04:05:37.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:36 smithi098 ceph-mon[88407]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-12T04:05:37.485 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (7m) 112s ago 29m 27.6M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (7m) 9s ago 29m 90.0M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (7m) 112s ago 28m 62.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (9m) 9s ago 34m 452M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (18m) 112s ago 35m 508M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (5m) 112s ago 36m 50.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:05:38.126 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (6m) 9s ago 34m 47.5M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (5m) 112s ago 34m 42.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (8m) 112s ago 30m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (8m) 9s ago 30m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (4m) 112s ago 33m 44.9M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (3m) 112s ago 33m 44.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (2m) 112s ago 33m 49.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (116s) 112s ago 32m 11.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (82s) 9s ago 32m 46.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (48s) 9s ago 32m 43.5M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (14s) 9s ago 31m 11.6M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 23a1ba5d3665 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (31m) 9s ago 31m 61.6M 3248M 17.2.0 e1d6a67b021e aaa5136875dc 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (8m) 9s ago 30m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (29m) 112s ago 29m 78.4M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:05:38.127 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (28m) 9s ago 28m 82.0M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-12T04:05:38.518 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 7 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 12 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:05:38.519 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:05:38.847 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:05:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:05:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:05:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-12T04:05:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:05:38.847 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "7/8 daemons upgraded", 2024-03-12T04:05:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading osd daemons", 2024-03-12T04:05:38.848 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:05:38.848 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[88407]: from='client.64480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[88407]: from='client.54552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[88407]: from='client.54558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[88407]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1361980073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[86021]: from='client.64480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[86021]: from='client.54552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[86021]: from='client.54558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[86021]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-12T04:05:39.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:39 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1361980073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:39 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:39 smithi153 ceph-mon[67960]: from='client.64480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:39 smithi153 ceph-mon[67960]: from='client.54552 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:39 smithi153 ceph-mon[67960]: from='client.54558 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:39 smithi153 ceph-mon[67960]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 964 B/s rd, 0 op/s 2024-03-12T04:05:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:39 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1361980073' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:05:40.475 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:40 smithi098 ceph-mon[86021]: from='client.54570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:40.475 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:40 smithi098 ceph-mon[88407]: from='client.54570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:40.476 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:05:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:05:40] "GET /metrics HTTP/1.1" 200 38281 "" "Prometheus/2.43.0" 2024-03-12T04:05:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:40 smithi153 ceph-mon[67960]: from='client.54570 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:05:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:41 smithi098 ceph-mon[88407]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T04:05:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:41.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:41 smithi098 ceph-mon[86021]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T04:05:41.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:41.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:41 smithi153 ceph-mon[67960]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-03-12T04:05:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:41.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:42 smithi153 ceph-mon[67960]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:05:42.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:42.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:42 smithi098 ceph-mon[86021]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:05:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:42.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:42 smithi098 ceph-mon[88407]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-12T04:05:45.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:45 smithi098 ceph-mon[86021]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:45.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:45 smithi098 ceph-mon[88407]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:45 smithi153 ceph-mon[67960]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:47.563 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:47 smithi153 ceph-mon[67960]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:47.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:47 smithi098 ceph-mon[86021]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:47.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:47 smithi098 ceph-mon[88407]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:48.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:48 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:48.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:48 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:48.610 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:48 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[86021]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[86021]: Upgrade: osd.7 is safe to restart 2024-03-12T04:05:49.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[86021]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[88407]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:49.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[88407]: Upgrade: osd.7 is safe to restart 2024-03-12T04:05:49.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:49 smithi098 ceph-mon[88407]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:49 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:49 smithi153 ceph-mon[67960]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-12T04:05:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:49 smithi153 ceph-mon[67960]: Upgrade: osd.7 is safe to restart 2024-03-12T04:05:49.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:49 smithi153 ceph-mon[67960]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:50.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:05:50] "GET /metrics HTTP/1.1" 200 38289 "" "Prometheus/2.43.0" 2024-03-12T04:05:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[86021]: Upgrade: Updating osd.7 2024-03-12T04:05:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T04:05:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[86021]: Deploying daemon osd.7 on smithi153 2024-03-12T04:05:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[86021]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[88407]: Upgrade: Updating osd.7 2024-03-12T04:05:50.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:50.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T04:05:50.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:50.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[88407]: Deploying daemon osd.7 on smithi153 2024-03-12T04:05:50.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:50 smithi098 ceph-mon[88407]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:50 smithi153 ceph-mon[67960]: Upgrade: Updating osd.7 2024-03-12T04:05:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:50.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-12T04:05:50.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:05:50.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:50 smithi153 ceph-mon[67960]: Deploying daemon osd.7 on smithi153 2024-03-12T04:05:50.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:50 smithi153 ceph-mon[67960]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:05:52.815 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:52 smithi153 systemd[1]: Stopping Ceph osd.7 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:05:53.192 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[37784]: 2024-03-12T04:05:52.812+0000 7f5f507fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:05:53.192 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[37784]: 2024-03-12T04:05:52.812+0000 7f5f507fa700 -1 osd.7 148 *** Got signal Terminated *** 2024-03-12T04:05:53.192 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:52 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[37784]: 2024-03-12T04:05:52.812+0000 7f5f507fa700 -1 osd.7 148 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:05:53.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:53 smithi098 ceph-mon[86021]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:53.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:53 smithi098 ceph-mon[86021]: osd.7 marked itself down and dead 2024-03-12T04:05:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:53 smithi098 ceph-mon[88407]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:53 smithi098 ceph-mon[88407]: osd.7 marked itself down and dead 2024-03-12T04:05:53.650 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:53 smithi153 ceph-mon[67960]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:05:53.650 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:53 smithi153 ceph-mon[67960]: osd.7 marked itself down and dead 2024-03-12T04:05:53.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:53 smithi153 podman[91685]: 2024-03-12 04:05:53.649204343 +0000 UTC m=+0.934368647 container died aaa5136875dcfe1eb5d771ea1746c11ede4348160324789467295996a067176c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, GIT_CLEAN=True, io.buildah.version=1.19.8, RELEASE=HEAD, io.openshift.expose-services=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, vcs-type=git, architecture=x86_64, name=centos-stream, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc.) 2024-03-12T04:05:54.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:54 smithi153 ceph-mon[67960]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:05:54.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:54 smithi153 ceph-mon[67960]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-12T04:05:54.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:54 smithi153 ceph-mon[67960]: osdmap e149: 8 total, 7 up, 8 in 2024-03-12T04:05:54.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:54 smithi153 podman[91685]: 2024-03-12 04:05:54.137073641 +0000 UTC m=+1.422237945 container cleanup aaa5136875dcfe1eb5d771ea1746c11ede4348160324789467295996a067176c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.expose-services=, release=754, ceph=True, vcs-type=git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-12T04:05:54.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:54 smithi153 bash[91685]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7 2024-03-12T04:05:54.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:54 smithi098 ceph-mon[86021]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:05:54.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:54 smithi098 ceph-mon[86021]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-12T04:05:54.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:54 smithi098 ceph-mon[86021]: osdmap e149: 8 total, 7 up, 8 in 2024-03-12T04:05:54.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:54 smithi098 ceph-mon[88407]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-12T04:05:54.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:54 smithi098 ceph-mon[88407]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-12T04:05:54.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:54 smithi098 ceph-mon[88407]: osdmap e149: 8 total, 7 up, 8 in 2024-03-12T04:05:54.799 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:54 smithi153 podman[91703]: 2024-03-12 04:05:54.455010636 +0000 UTC m=+0.796165195 container remove aaa5136875dcfe1eb5d771ea1746c11ede4348160324789467295996a067176c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_CLEAN=True, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, architecture=x86_64, ceph=True, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container) 2024-03-12T04:05:55.077 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:54 smithi153 podman[91795]: 2024-03-12 04:05:54.797629126 +0000 UTC m=+0.103919438 container create 403d00e5e27f23b18338b3630e75ead8eb47a5cac8d603275d836a12a4bed574 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux ) 2024-03-12T04:05:55.077 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:54 smithi153 podman[91795]: 2024-03-12 04:05:54.711549342 +0000 UTC m=+0.017839657 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:05:55.077 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 podman[91795]: 2024-03-12 04:05:55.000196649 +0000 UTC m=+0.306486977 container init 403d00e5e27f23b18338b3630e75ead8eb47a5cac8d603275d836a12a4bed574 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:05:55.077 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 podman[91795]: 2024-03-12 04:05:55.00341854 +0000 UTC m=+0.309708852 container start 403d00e5e27f23b18338b3630e75ead8eb47a5cac8d603275d836a12a4bed574 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:05:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:55 smithi153 ceph-mon[67960]: pgmap v218: 161 pgs: 9 peering, 20 stale+active+clean, 132 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T04:05:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:55 smithi153 ceph-mon[67960]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:55.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:55 smithi153 ceph-mon[67960]: osdmap e150: 8 total, 7 up, 8 in 2024-03-12T04:05:55.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 podman[91795]: 2024-03-12 04:05:55.075177228 +0000 UTC m=+0.381467539 container attach 403d00e5e27f23b18338b3630e75ead8eb47a5cac8d603275d836a12a4bed574 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-12T04:05:55.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 podman[91795]: 2024-03-12 04:05:55.30595418 +0000 UTC m=+0.612244491 container died 403d00e5e27f23b18338b3630e75ead8eb47a5cac8d603275d836a12a4bed574 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-03-12T04:05:55.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:55 smithi098 ceph-mon[86021]: pgmap v218: 161 pgs: 9 peering, 20 stale+active+clean, 132 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T04:05:55.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:55 smithi098 ceph-mon[86021]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:55.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:55 smithi098 ceph-mon[86021]: osdmap e150: 8 total, 7 up, 8 in 2024-03-12T04:05:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:55 smithi098 ceph-mon[88407]: pgmap v218: 161 pgs: 9 peering, 20 stale+active+clean, 132 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-12T04:05:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:55 smithi098 ceph-mon[88407]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-12T04:05:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:55 smithi098 ceph-mon[88407]: osdmap e150: 8 total, 7 up, 8 in 2024-03-12T04:05:55.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 podman[91821]: 2024-03-12 04:05:55.786810607 +0000 UTC m=+0.474819731 container remove 403d00e5e27f23b18338b3630e75ead8eb47a5cac8d603275d836a12a4bed574 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-12T04:05:55.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-12T04:05:55.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 systemd[1]: Stopped Ceph osd.7 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:05:55.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:55 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.7.service: Consumed 16.158s CPU time. 2024-03-12T04:05:56.375 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 systemd[1]: Starting Ceph osd.7 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:05:56.375 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 podman[91921]: 2024-03-12 04:05:56.127055211 +0000 UTC m=+0.017756788 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:05:56.375 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 podman[91921]: 2024-03-12 04:05:56.246427867 +0000 UTC m=+0.137129427 container create 3ec5679635d98e8444471a72e4649916c019f649681c7769dd427c6eae5e942c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-12T04:05:56.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 podman[91921]: 2024-03-12 04:05:56.507786944 +0000 UTC m=+0.398488523 container init 3ec5679635d98e8444471a72e4649916c019f649681c7769dd427c6eae5e942c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-12T04:05:56.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 podman[91921]: 2024-03-12 04:05:56.511068481 +0000 UTC m=+0.401770043 container start 3ec5679635d98e8444471a72e4649916c019f649681c7769dd427c6eae5e942c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T04:05:56.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 podman[91921]: 2024-03-12 04:05:56.599202375 +0000 UTC m=+0.489903947 container attach 3ec5679635d98e8444471a72e4649916c019f649681c7769dd427c6eae5e942c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-12T04:05:57.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate[91936]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-12T04:05:57.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate[91936]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T04:05:57.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 bash[91921]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-12T04:05:57.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 bash[91921]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-12T04:05:57.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 bash[91921]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T04:05:57.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 bash[91921]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T04:05:57.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 bash[91921]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 bash[91921]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 bash[91921]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate[91936]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate[91936]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate[91936]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate[91936]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate[91936]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-12T04:05:57.242 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:56 smithi153 podman[91921]: 2024-03-12 04:05:56.979287031 +0000 UTC m=+0.869988599 container died 3ec5679635d98e8444471a72e4649916c019f649681c7769dd427c6eae5e942c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=main-2b0101e) 2024-03-12T04:05:57.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:57 smithi098 ceph-mon[86021]: pgmap v220: 161 pgs: 21 active+undersized+degraded, 9 peering, 5 stale+active+clean, 28 active+undersized, 98 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76/723 objects degraded (10.512%) 2024-03-12T04:05:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:57.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:57 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:57 smithi098 ceph-mon[88407]: pgmap v220: 161 pgs: 21 active+undersized+degraded, 9 peering, 5 stale+active+clean, 28 active+undersized, 98 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76/723 objects degraded (10.512%) 2024-03-12T04:05:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:57 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:57.662 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:57 smithi153 ceph-mon[67960]: pgmap v220: 161 pgs: 21 active+undersized+degraded, 9 peering, 5 stale+active+clean, 28 active+undersized, 98 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76/723 objects degraded (10.512%) 2024-03-12T04:05:57.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:57.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:57 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:05:57.663 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 podman[92069]: 2024-03-12 04:05:57.477687267 +0000 UTC m=+0.492361612 container remove 3ec5679635d98e8444471a72e4649916c019f649681c7769dd427c6eae5e942c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-12T04:05:57.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 podman[92135]: 2024-03-12 04:05:57.661700166 +0000 UTC m=+0.087506159 container create 782d8be774e8d57fdb4e6315efd54de04389065c5c1f56a5fb2eee5784bb83a3 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-12T04:05:57.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 podman[92135]: 2024-03-12 04:05:57.591976662 +0000 UTC m=+0.017782656 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:05:57.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 podman[92135]: 2024-03-12 04:05:57.863870834 +0000 UTC m=+0.289676828 container init 782d8be774e8d57fdb4e6315efd54de04389065c5c1f56a5fb2eee5784bb83a3 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:05:57.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 podman[92135]: 2024-03-12 04:05:57.866766848 +0000 UTC m=+0.292572845 container start 782d8be774e8d57fdb4e6315efd54de04389065c5c1f56a5fb2eee5784bb83a3 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-12T04:05:57.943 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 ceph-osd[92154]: -- 172.21.15.153:0/731676826 <== mon.2 v2:172.21.15.153:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e0769af680 con 0x55e076978c00 2024-03-12T04:05:58.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 bash[92135]: 782d8be774e8d57fdb4e6315efd54de04389065c5c1f56a5fb2eee5784bb83a3 2024-03-12T04:05:58.241 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:57 smithi153 systemd[1]: Started Ceph osd.7 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:05:58.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:58 smithi098 ceph-mon[86021]: Health check failed: Degraded data redundancy: 76/723 objects degraded (10.512%), 21 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:58.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:58 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:58.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:58 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:58 smithi098 ceph-mon[88407]: Health check failed: Degraded data redundancy: 76/723 objects degraded (10.512%), 21 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:58 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:58.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:58 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:58 smithi153 ceph-mon[67960]: Health check failed: Degraded data redundancy: 76/723 objects degraded (10.512%), 21 pgs degraded (PG_DEGRADED) 2024-03-12T04:05:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:58 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:58 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:05:58.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:05:58 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:05:58.406+0000 7fc4b64187c0 -1 Falling back to public interface 2024-03-12T04:05:59.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:59 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:59.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:05:59 smithi098 ceph-mon[86021]: pgmap v221: 161 pgs: 21 active+undersized+degraded, 9 peering, 5 stale+active+clean, 28 active+undersized, 98 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76/723 objects degraded (10.512%) 2024-03-12T04:05:59.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:59 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:59.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:05:59 smithi098 ceph-mon[88407]: pgmap v221: 161 pgs: 21 active+undersized+degraded, 9 peering, 5 stale+active+clean, 28 active+undersized, 98 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76/723 objects degraded (10.512%) 2024-03-12T04:05:59.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:59 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:05:59.663 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:05:59 smithi153 ceph-mon[67960]: pgmap v221: 161 pgs: 21 active+undersized+degraded, 9 peering, 5 stale+active+clean, 28 active+undersized, 98 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 76/723 objects degraded (10.512%) 2024-03-12T04:06:00.549 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:00 smithi153 ceph-mon[67960]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-12T04:06:00.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:00 smithi098 ceph-mon[86021]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-12T04:06:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:06:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:06:00] "GET /metrics HTTP/1.1" 200 38372 "" "Prometheus/2.43.0" 2024-03-12T04:06:00.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:00 smithi098 ceph-mon[88407]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-12T04:06:01.261 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:06:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:06:01.143+0000 7fc4b64187c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-12T04:06:01.261 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:06:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:06:01.172+0000 7fc4b64187c0 -1 osd.7 148 log_to_monitors true 2024-03-12T04:06:01.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:01 smithi098 ceph-mon[88407]: pgmap v222: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 94/723 objects degraded (13.001%) 2024-03-12T04:06:01.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:01 smithi098 ceph-mon[88407]: from='osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T04:06:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:01 smithi098 ceph-mon[88407]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T04:06:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:01 smithi098 ceph-mon[86021]: pgmap v222: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 94/723 objects degraded (13.001%) 2024-03-12T04:06:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:01 smithi098 ceph-mon[86021]: from='osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T04:06:01.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:01 smithi098 ceph-mon[86021]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T04:06:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:01 smithi153 ceph-mon[67960]: pgmap v222: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 94/723 objects degraded (13.001%) 2024-03-12T04:06:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:01 smithi153 ceph-mon[67960]: from='osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T04:06:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:01 smithi153 ceph-mon[67960]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-12T04:06:02.192 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:06:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:06:01.792+0000 7fc4a3868700 -1 osd.7 148 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-12T04:06:02.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[86021]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[86021]: from='osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[86021]: osdmap e151: 8 total, 7 up, 8 in 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[86021]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[86021]: from='osd.7 ' entity='osd.7' 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[88407]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[88407]: from='osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[88407]: osdmap e151: 8 total, 7 up, 8 in 2024-03-12T04:06:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[88407]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:06:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[88407]: from='osd.7 ' entity='osd.7' 2024-03-12T04:06:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:02.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:02 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:02.587 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:02 smithi153 ceph-mon[67960]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-12T04:06:02.587 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:02 smithi153 ceph-mon[67960]: from='osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:06:02.587 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:02 smithi153 ceph-mon[67960]: osdmap e151: 8 total, 7 up, 8 in 2024-03-12T04:06:02.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:02 smithi153 ceph-mon[67960]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-03-12T04:06:02.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:02 smithi153 ceph-mon[67960]: from='osd.7 ' entity='osd.7' 2024-03-12T04:06:02.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:02 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:02.588 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:02 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[86021]: pgmap v224: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 94/723 objects degraded (13.001%) 2024-03-12T04:06:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[86021]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[86021]: osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154] boot 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[86021]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[88407]: pgmap v224: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 94/723 objects degraded (13.001%) 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[88407]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[88407]: osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154] boot 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[88407]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T04:06:03.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:03.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:03 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:03 smithi153 ceph-mon[67960]: pgmap v224: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 94/723 objects degraded (13.001%) 2024-03-12T04:06:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:03 smithi153 ceph-mon[67960]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-12T04:06:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:03 smithi153 ceph-mon[67960]: osd.7 [v2:172.21.15.153:6824/3019544154,v1:172.21.15.153:6825/3019544154] boot 2024-03-12T04:06:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:03 smithi153 ceph-mon[67960]: osdmap e152: 8 total, 8 up, 8 in 2024-03-12T04:06:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:03 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-12T04:06:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:03 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:03 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:04.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:04 smithi153 ceph-mon[67960]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T04:06:04.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:04 smithi153 ceph-mon[67960]: pgmap v227: 161 pgs: 13 peering, 22 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/723 objects degraded (11.618%) 2024-03-12T04:06:05.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:04 smithi098 ceph-mon[88407]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T04:06:05.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:04 smithi098 ceph-mon[88407]: pgmap v227: 161 pgs: 13 peering, 22 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/723 objects degraded (11.618%) 2024-03-12T04:06:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:04 smithi098 ceph-mon[86021]: osdmap e153: 8 total, 8 up, 8 in 2024-03-12T04:06:05.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:04 smithi098 ceph-mon[86021]: pgmap v227: 161 pgs: 13 peering, 22 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 245 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 84/723 objects degraded (11.618%) 2024-03-12T04:06:06.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:05 smithi098 ceph-mon[86021]: Health check update: Degraded data redundancy: 84/723 objects degraded (11.618%), 22 pgs degraded (PG_DEGRADED) 2024-03-12T04:06:06.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:05 smithi098 ceph-mon[88407]: Health check update: Degraded data redundancy: 84/723 objects degraded (11.618%), 22 pgs degraded (PG_DEGRADED) 2024-03-12T04:06:06.145 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:05 smithi153 ceph-mon[67960]: Health check update: Degraded data redundancy: 84/723 objects degraded (11.618%), 22 pgs degraded (PG_DEGRADED) 2024-03-12T04:06:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: pgmap v228: 161 pgs: 13 peering, 9 active+undersized+degraded, 13 active+undersized, 126 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 41/723 objects degraded (5.671%) 2024-03-12T04:06:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-12T04:06:07.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: pgmap v228: 161 pgs: 13 peering, 9 active+undersized+degraded, 13 active+undersized, 126 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 41/723 objects degraded (5.671%) 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-12T04:06:07.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-12T04:06:07.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:06 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-12T04:06:07.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: pgmap v228: 161 pgs: 13 peering, 9 active+undersized+degraded, 13 active+undersized, 126 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 41/723 objects degraded (5.671%) 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:07.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-12T04:06:07.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-12T04:06:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-12T04:06:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-12T04:06:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-12T04:06:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-12T04:06:07.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:06 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-12T04:06:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all osd 2024-03-12T04:06:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: Upgrade: Setting require_osd_release to 19 squid 2024-03-12T04:06:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-12T04:06:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-12T04:06:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T04:06:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all osd 2024-03-12T04:06:08.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: Upgrade: Setting require_osd_release to 19 squid 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:06:08.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:08.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:06:08.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:06:08.081 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:07 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:06:08.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all osd 2024-03-12T04:06:08.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: Upgrade: Setting require_osd_release to 19 squid 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: osdmap e154: 8 total, 8 up, 8 in 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:06:08.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:06:08.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:07 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mds 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nfs 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: Upgrade: Finalizing container_image settings 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:06:09.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: Upgrade: Complete! 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:09.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[86021]: pgmap v230: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mds 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nfs 2024-03-12T04:06:09.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: Upgrade: Finalizing container_image settings 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: Upgrade: Complete! 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:09.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:08 smithi098 ceph-mon[88407]: pgmap v230: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mds 2024-03-12T04:06:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:06:09.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nfs 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: Upgrade: Finalizing container_image settings 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: Upgrade: Complete! 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:09.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:08 smithi153 ceph-mon[67960]: pgmap v230: 161 pgs: 13 peering, 148 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:09.902 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T04:06:10.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:09 smithi098 ceph-mon[88407]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 9 pgs degraded) 2024-03-12T04:06:10.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:09 smithi098 ceph-mon[88407]: Cluster is now healthy 2024-03-12T04:06:10.073 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:09 smithi098 ceph-mon[88407]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:09 smithi098 ceph-mon[86021]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 9 pgs degraded) 2024-03-12T04:06:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:09 smithi098 ceph-mon[86021]: Cluster is now healthy 2024-03-12T04:06:10.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:09 smithi098 ceph-mon[86021]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:09 smithi153 ceph-mon[67960]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 9 pgs degraded) 2024-03-12T04:06:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:09 smithi153 ceph-mon[67960]: Cluster is now healthy 2024-03-12T04:06:10.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:09 smithi153 ceph-mon[67960]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:10.328 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:06:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:06:10] "GET /metrics HTTP/1.1" 200 38372 "" "Prometheus/2.43.0" 2024-03-12T04:06:11.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:10 smithi098 ceph-mon[88407]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2024-03-12T04:06:11.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:10 smithi098 ceph-mon[86021]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2024-03-12T04:06:11.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:10 smithi153 ceph-mon[67960]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2024-03-12T04:06:11.835 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:06:11.835 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (8m) 2m ago 30m 27.6M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:06:11.835 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (8m) 9s ago 30m 89.6M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:06:11.835 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (7m) 2m ago 29m 62.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:06:11.835 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (9m) 9s ago 34m 452M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (19m) 2m ago 36m 508M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (6m) 2m ago 36m 50.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (7m) 9s ago 35m 49.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (6m) 2m ago 35m 42.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (9m) 2m ago 31m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (9m) 9s ago 31m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (4m) 2m ago 34m 44.9M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (4m) 2m ago 34m 44.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (3m) 2m ago 33m 49.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (2m) 2m ago 33m 11.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (116s) 9s ago 33m 46.9M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:06:11.836 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (82s) 9s ago 32m 43.9M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:06:11.837 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (48s) 9s ago 32m 49.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 23a1ba5d3665 2024-03-12T04:06:11.837 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (14s) 9s ago 31m 12.7M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 782d8be774e8 2024-03-12T04:06:11.837 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (8m) 9s ago 31m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:06:11.837 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (29m) 2m ago 29m 78.4M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:06:11.837 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (29m) 9s ago 29m 82.2M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:06:12.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:12.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:12.533 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:12 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:12.533 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:12.533 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:12.533 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:12 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:12.622 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-03-12T04:06:12.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:12.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:12.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:12 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:13.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:13 smithi098 ceph-mon[86021]: from='client.64522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:13.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:13 smithi098 ceph-mon[86021]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:13 smithi098 ceph-mon[88407]: from='client.64522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:13.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:13 smithi098 ceph-mon[88407]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:13 smithi153 ceph-mon[67960]: from='client.64522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:13 smithi153 ceph-mon[67960]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:14.710 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:06:15.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:15 smithi098 ceph-mon[86021]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:06:15.323 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:15 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/895221104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:15.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:15 smithi098 ceph-mon[88407]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:06:15.324 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:15 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/895221104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:15.496 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-03-12T04:06:15.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:15 smithi153 ceph-mon[67960]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:06:15.510 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:15 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/895221104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:17.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:17 smithi098 ceph-mon[86021]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:06:17.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:17 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:17.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:17 smithi098 ceph-mon[88407]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:06:17.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:17 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:17.683 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)" 2024-03-12T04:06:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:17 smithi153 ceph-mon[67960]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:06:17.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:17 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:18.323 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:18 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2152441997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:18.324 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:18 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2152441997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:18.463 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T04:06:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:18 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2152441997' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:19.437 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:19 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:19.437 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:19 smithi098 ceph-mon[86021]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-12T04:06:19.438 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:19 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:19.438 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:19 smithi098 ceph-mon[88407]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-12T04:06:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:19 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:19 smithi153 ceph-mon[67960]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-12T04:06:20.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:06:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:06:20] "GET /metrics HTTP/1.1" 200 38372 "" "Prometheus/2.43.0" 2024-03-12T04:06:20.479 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:06:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": null, 2024-03-12T04:06:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": false, 2024-03-12T04:06:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "", 2024-03-12T04:06:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:06:20.479 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": null, 2024-03-12T04:06:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T04:06:20.480 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:06:20.480 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:06:21.250 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T04:06:21.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:21 smithi098 ceph-mon[86021]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:21.563 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:21 smithi098 ceph-mon[88407]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:21.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:21 smithi153 ceph-mon[67960]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:22.545 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:22 smithi098 ceph-mon[88407]: from='client.54606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:22.545 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:22 smithi098 ceph-mon[86021]: from='client.54606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:22.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:22 smithi153 ceph-mon[67960]: from='client.54606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:23.302 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T04:06:23.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:23 smithi098 ceph-mon[86021]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:23.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:23 smithi098 ceph-mon[88407]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:23.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:23 smithi153 ceph-mon[67960]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:23.989 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-03-12T04:06:24.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:24 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2686247472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:06:24.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:24 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2686247472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:06:24.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:24 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2686247472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:06:25.543 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:25 smithi098 ceph-mon[86021]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:25.543 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:25 smithi098 ceph-mon[88407]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:25 smithi153 ceph-mon[67960]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:26.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:26.550 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:26.550 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:27.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:27 smithi098 ceph-mon[86021]: from='client.54618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:27.446 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:27 smithi098 ceph-mon[86021]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:27.447 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:27 smithi098 ceph-mon[88407]: from='client.54618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:27.447 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:27 smithi098 ceph-mon[88407]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:27.509 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:06:27.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:27 smithi153 ceph-mon[67960]: from='client.54618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:27.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:27 smithi153 ceph-mon[67960]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:28.191 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[88407]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:28.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:06:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:06:28.825 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:28 smithi098 ceph-mon[86021]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:06:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:28.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:06:28.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:28 smithi153 ceph-mon[67960]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:06:30] "GET /metrics HTTP/1.1" 200 38371 "" "Prometheus/2.43.0" 2024-03-12T04:06:30.928 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mgr 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mon 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all crash 2024-03-12T04:06:30.929 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all osd 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mds 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[86021]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:06:30.930 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mgr 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mon 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all crash 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all osd 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.931 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mds 2024-03-12T04:06:30.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.932 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:30 smithi098 ceph-mon[88407]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:30.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:06:30.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:06:30.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:30.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mgr 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mon 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all crash 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all osd 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:30.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mds 2024-03-12T04:06:30.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:30.944 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:30 smithi153 ceph-mon[67960]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:32.348 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (8m) 2m ago 30m 27.6M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (8m) 30s ago 30m 89.6M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (8m) 2m ago 29m 62.5M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (9m) 30s ago 35m 452M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (19m) 2m ago 36m 508M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (6m) 2m ago 36m 50.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (7m) 30s ago 35m 49.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (6m) 2m ago 35m 42.3M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (9m) 2m ago 31m 15.0M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:06:33.012 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (9m) 30s ago 31m 15.1M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (5m) 2m ago 34m 44.9M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (4m) 2m ago 34m 44.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (3m) 2m ago 34m 49.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (2m) 2m ago 33m 11.6M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (2m) 30s ago 33m 46.9M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (103s) 30s ago 32m 43.9M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (69s) 30s ago 32m 49.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 23a1ba5d3665 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (35s) 30s ago 32m 12.7M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 782d8be774e8 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (9m) 30s ago 31m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (29m) 2m ago 29m 78.4M - 17.2.0 e1d6a67b021e a865891aaa4e 2024-03-12T04:06:33.013 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (29m) 30s ago 29m 82.2M - 17.2.0 e1d6a67b021e 4f0c7c56693e 2024-03-12T04:06:33.428 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 8 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-12T04:06:33.429 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:06:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:06:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-12T04:06:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 13 2024-03-12T04:06:33.430 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:06:33.430 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:06:33.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[86021]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:33.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[86021]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:33.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:33.430 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:06:33.431 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:33.431 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[88407]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:33.431 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[88407]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:33.431 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:33.431 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:06:33.431 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:33 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:33 smithi153 ceph-mon[67960]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:33 smithi153 ceph-mon[67960]: from='client.64558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi098.tgzbnw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:06:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:33 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "0/2 daemons upgraded", 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading rgw daemons", 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:06:33.763 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:06:34.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[86021]: Upgrade: Updating rgw.foo.smithi098.tgzbnw (1/2) 2024-03-12T04:06:34.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[86021]: Deploying daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T04:06:34.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[86021]: from='client.64564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:34.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[86021]: from='client.64570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:34.420 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/3796565052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:34.420 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[88407]: Upgrade: Updating rgw.foo.smithi098.tgzbnw (1/2) 2024-03-12T04:06:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[88407]: Deploying daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T04:06:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[88407]: from='client.64564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[88407]: from='client.64570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:34.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:34 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/3796565052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:34 smithi153 ceph-mon[67960]: Upgrade: Updating rgw.foo.smithi098.tgzbnw (1/2) 2024-03-12T04:06:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:34 smithi153 ceph-mon[67960]: Deploying daemon rgw.foo.smithi098.tgzbnw on smithi098 2024-03-12T04:06:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:34 smithi153 ceph-mon[67960]: from='client.64564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:34 smithi153 ceph-mon[67960]: from='client.64570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:34 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/3796565052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:35.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:35 smithi098 ceph-mon[86021]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:35.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:35 smithi098 ceph-mon[86021]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:35.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:35 smithi098 ceph-mon[88407]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:35.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:35 smithi098 ceph-mon[88407]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:35 smithi153 ceph-mon[67960]: from='client.54648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:06:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:35 smithi153 ceph-mon[67960]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:06:37.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:37 smithi098 ceph-mon[86021]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:37 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:37.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:37 smithi098 ceph-mon[88407]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:37.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:37.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:37 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:37.605 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:37 smithi153 ceph-mon[67960]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:06:37.606 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:37.606 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:37 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:39 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:39 smithi153 ceph-mon[67960]: Upgrade: Updating rgw.foo.smithi153.tuboyg (2/2) 2024-03-12T04:06:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:06:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:39 smithi153 ceph-mon[67960]: Deploying daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T04:06:39.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:39 smithi153 ceph-mon[67960]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:39.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[86021]: Upgrade: Updating rgw.foo.smithi153.tuboyg (2/2) 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[86021]: Deploying daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[86021]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[88407]: Upgrade: Updating rgw.foo.smithi153.tuboyg (2/2) 2024-03-12T04:06:39.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:39.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi153.tuboyg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-12T04:06:39.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:39.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[88407]: Deploying daemon rgw.foo.smithi153.tuboyg on smithi153 2024-03-12T04:06:39.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:39 smithi098 ceph-mon[88407]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 249 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:06:40.475 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:06:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:06:40] "GET /metrics HTTP/1.1" 200 38371 "" "Prometheus/2.43.0" 2024-03-12T04:06:41.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:41 smithi098 ceph-mon[86021]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2024-03-12T04:06:41.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:41 smithi098 ceph-mon[88407]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2024-03-12T04:06:41.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:41.628 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:41 smithi153 ceph-mon[67960]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 253 MiB used, 715 GiB / 715 GiB avail; 31 KiB/s rd, 0 B/s wr, 49 op/s 2024-03-12T04:06:41.628 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:43.105 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:43.106 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:43.106 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:42 smithi153 ceph-mon[67960]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 85 B/s wr, 170 op/s 2024-03-12T04:06:43.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:43.140 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:43.141 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:42 smithi098 ceph-mon[86021]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 85 B/s wr, 170 op/s 2024-03-12T04:06:43.141 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:43.141 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:43.141 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:42 smithi098 ceph-mon[88407]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 107 KiB/s rd, 85 B/s wr, 170 op/s 2024-03-12T04:06:45.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:45 smithi098 ceph-mon[86021]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 85 B/s wr, 206 op/s 2024-03-12T04:06:45.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:45 smithi098 ceph-mon[88407]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 85 B/s wr, 206 op/s 2024-03-12T04:06:45.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:45 smithi153 ceph-mon[67960]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s rd, 85 B/s wr, 206 op/s 2024-03-12T04:06:46.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[86021]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 170 B/s wr, 288 op/s 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:46 smithi098 ceph-mon[88407]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 170 B/s wr, 288 op/s 2024-03-12T04:06:46.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:46 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:46 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:46 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:46 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:46.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:46 smithi153 ceph-mon[67960]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 181 KiB/s rd, 170 B/s wr, 288 op/s 2024-03-12T04:06:47.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.722 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.723 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.723 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.723 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:47 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:47.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:47 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:48 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:48.693 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:48 smithi153 ceph-mon[67960]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 170 B/s wr, 307 op/s 2024-03-12T04:06:48.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:48 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:48.750 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:48 smithi098 ceph-mon[86021]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 170 B/s wr, 307 op/s 2024-03-12T04:06:48.750 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:48 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:48.750 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:48 smithi098 ceph-mon[88407]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 170 B/s wr, 307 op/s 2024-03-12T04:06:50.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:06:50] "GET /metrics HTTP/1.1" 200 38375 "" "Prometheus/2.43.0" 2024-03-12T04:06:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 170 B/s wr, 307 op/s 2024-03-12T04:06:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.tgzbnw"}]: dispatch 2024-03-12T04:06:51.075 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.tgzbnw"}]': finished 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.tuboyg"}]: dispatch 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.tuboyg"}]': finished 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 170 B/s wr, 307 op/s 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:51.077 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.tgzbnw"}]: dispatch 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.tgzbnw"}]': finished 2024-03-12T04:06:51.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.tuboyg"}]: dispatch 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.tuboyg"}]': finished 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:06:51.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:06:51.081 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:50 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:06:51.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 193 KiB/s rd, 170 B/s wr, 307 op/s 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.tgzbnw"}]: dispatch 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi098.tgzbnw"}]': finished 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.tuboyg"}]: dispatch 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi153.tuboyg"}]': finished 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-12T04:06:51.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:51.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-12T04:06:51.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:50 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all rgw 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nfs 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Upgrade: Finalizing container_image settings 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:06:52.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:06:52.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all rgw 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nfs 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Upgrade: Finalizing container_image settings 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:06:52.076 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:06:52.077 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Upgrade: Complete! 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:06:52.078 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: Checking dashboard <-> RGW credentials 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.079 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Upgrade: Complete! 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: Checking dashboard <-> RGW credentials 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.080 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:51 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all rgw 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nfs 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all nvmeof 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Upgrade: Finalizing container_image settings 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-12T04:06:52.193 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-12T04:06:52.194 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Upgrade: Complete! 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: Checking dashboard <-> RGW credentials 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:06:52.195 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:06:52.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:52.196 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:51 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:06:53.073 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:52 smithi098 ceph-mon[86021]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 170 B/s wr, 259 op/s 2024-03-12T04:06:53.074 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:52 smithi098 ceph-mon[88407]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 170 B/s wr, 259 op/s 2024-03-12T04:06:53.192 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:52 smithi153 ceph-mon[67960]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 170 B/s wr, 259 op/s 2024-03-12T04:06:55.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:55 smithi098 ceph-mon[86021]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 85 B/s wr, 138 op/s 2024-03-12T04:06:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:55 smithi098 ceph-mon[88407]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 85 B/s wr, 138 op/s 2024-03-12T04:06:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:55 smithi153 ceph-mon[67960]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 85 B/s wr, 138 op/s 2024-03-12T04:06:56.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:56.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:06:57.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:57 smithi098 ceph-mon[86021]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 85 B/s wr, 104 op/s 2024-03-12T04:06:57.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:57 smithi098 ceph-mon[88407]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 85 B/s wr, 104 op/s 2024-03-12T04:06:57.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:57 smithi153 ceph-mon[67960]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 85 B/s wr, 104 op/s 2024-03-12T04:06:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:59 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:06:59 smithi153 ceph-mon[67960]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 22 op/s 2024-03-12T04:06:59.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:59 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:59.809 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:06:59 smithi098 ceph-mon[86021]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 22 op/s 2024-03-12T04:06:59.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:59 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:06:59.809 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:06:59 smithi098 ceph-mon[88407]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 0 B/s wr, 22 op/s 2024-03-12T04:07:00.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:07:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:07:00] "GET /metrics HTTP/1.1" 200 38407 "" "Prometheus/2.43.0" 2024-03-12T04:07:01.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:01 smithi153 ceph-mon[67960]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-12T04:07:01.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:01 smithi098 ceph-mon[86021]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-12T04:07:01.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:01 smithi098 ceph-mon[88407]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-12T04:07:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:03 smithi153 ceph-mon[67960]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-03-12T04:07:03.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:03 smithi098 ceph-mon[86021]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-03-12T04:07:03.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:03 smithi098 ceph-mon[88407]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-03-12T04:07:04.855 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T04:07:05.415 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:05 smithi098 ceph-mon[88407]: from='client.64663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:05.416 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:05 smithi098 ceph-mon[88407]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-12T04:07:05.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:05 smithi098 ceph-mon[86021]: from='client.64663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:05.416 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:05 smithi098 ceph-mon[86021]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-12T04:07:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:05 smithi153 ceph-mon[67960]: from='client.64663 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:05.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:05 smithi153 ceph-mon[67960]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-12T04:07:06.840 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (9m) 21s ago 31m 27.4M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (9m) 21s ago 30m 89.6M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (8m) 21s ago 30m 65.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (10m) 21s ago 35m 453M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (20m) 21s ago 37m 514M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (7m) 21s ago 37m 57.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (8m) 21s ago 36m 50.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (6m) 21s ago 36m 45.7M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (10m) 21s ago 32m 14.6M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (10m) 21s ago 31m 15.2M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:07:06.841 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (5m) 21s ago 35m 53.7M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (5m) 21s ago 35m 54.2M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (4m) 21s ago 34m 56.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (3m) 21s ago 34m 59.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (2m) 21s ago 33m 52.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (2m) 21s ago 33m 48.4M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (103s) 21s ago 33m 53.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 23a1ba5d3665 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (69s) 21s ago 32m 54.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 782d8be774e8 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (9m) 21s ago 32m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (30s) 21s ago 30m 78.5M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 12a7e0e5eaeb 2024-03-12T04:07:06.842 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (25s) 21s ago 30m 75.2M - 19.0.0-1979-g2b0101eb 4133f8bc41bb ed96ecebb071 2024-03-12T04:07:07.444 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-03-12T04:07:07.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:07 smithi153 ceph-mon[67960]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-03-12T04:07:07.748 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:07 smithi098 ceph-mon[86021]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-03-12T04:07:07.748 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:07 smithi098 ceph-mon[88407]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-03-12T04:07:08.461 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:08 smithi098 ceph-mon[88407]: from='client.54738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:08.461 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:08 smithi098 ceph-mon[86021]: from='client.54738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:08 smithi153 ceph-mon[67960]: from='client.54738 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:09.553 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:07:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:09 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:09.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:09 smithi153 ceph-mon[67960]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:09.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:09 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:09.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:09 smithi098 ceph-mon[86021]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:09 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:09.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:09 smithi098 ceph-mon[88407]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:10.201 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-03-12T04:07:10.508 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:10 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2821551284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:10.509 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:10 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2821551284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:10.509 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:07:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:07:10] "GET /metrics HTTP/1.1" 200 38407 "" "Prometheus/2.43.0" 2024-03-12T04:07:10.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:10 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2821551284' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:11.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:11 smithi098 ceph-mon[86021]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:11.618 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:11.618 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:11 smithi098 ceph-mon[88407]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:11.618 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:11 smithi153 ceph-mon[67960]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:12.158 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)" 2024-03-12T04:07:12.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:12 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/3977538859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:12.718 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:12 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/3977538859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:12.718 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:12 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/3977538859' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:12.748 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T04:07:13.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:13 smithi153 ceph-mon[67960]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:13.700 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:13 smithi098 ceph-mon[86021]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:13.700 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:13 smithi098 ceph-mon[88407]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:14.772 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:07:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": null, 2024-03-12T04:07:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": false, 2024-03-12T04:07:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "", 2024-03-12T04:07:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [], 2024-03-12T04:07:14.772 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": null, 2024-03-12T04:07:14.773 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "", 2024-03-12T04:07:14.773 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:07:14.773 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:07:15.483 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:15 smithi098 ceph-mon[86021]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:15.483 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:15 smithi098 ceph-mon[88407]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:15.507 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T04:07:15.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:15 smithi153 ceph-mon[67960]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:16.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:16 smithi098 ceph-mon[88407]: from='client.64687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:16.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:16 smithi098 ceph-mon[86021]: from='client.64687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:16.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:16 smithi153 ceph-mon[67960]: from='client.64687 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:17.491 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:17 smithi098 ceph-mon[86021]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:17.491 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:17 smithi098 ceph-mon[88407]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:17.620 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T04:07:17.653 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:17 smithi153 ceph-mon[67960]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:18.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-12T04:07:18.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:18 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2818465033' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:07:18.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:18 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2818465033' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:07:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:18 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2818465033' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:07:19.527 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:19 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:19.527 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:19 smithi098 ceph-mon[86021]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:19.527 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:19 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:19.527 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:19 smithi098 ceph-mon[88407]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:19 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:19.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:19 smithi153 ceph-mon[67960]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:20.036 INFO:teuthology.orchestra.run.smithi098.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:07:20.323 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:07:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:07:20] "GET /metrics HTTP/1.1" 200 38408 "" "Prometheus/2.43.0" 2024-03-12T04:07:20.815 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T04:07:20.826 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-12T04:07:20.827 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: from='client.54768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:07:21.074 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[86021]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: from='client.54768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:07:21.075 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:21 smithi098 ceph-mon[88407]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: from='client.54768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:07:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:21.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:07:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:07:21.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:21 smithi153 ceph-mon[67960]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:23.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:07:23.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:07:23.034 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mgr 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mon 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all crash 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all osd 2024-03-12T04:07:23.035 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all mds 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all rgw 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.036 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[86021]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:07:23.037 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mgr 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mon 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all crash 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all osd 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.038 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all mds 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all rgw 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.039 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:22 smithi098 ceph-mon[88407]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Target is version 19.0.0-1979-g2b0101eb (squid) 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, digests ['quay.ceph.io/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379'] 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mgr 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mon 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.443 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all crash 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all osd 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all mds 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all rgw 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all rbd-mirror 2024-03-12T04:07:23.444 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-12T04:07:23.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:23.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: Upgrade: Setting container_image for all ceph-exporter 2024-03-12T04:07:23.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:23.445 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:22 smithi153 ceph-mon[67960]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:25.036 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:07:25.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[86021]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:25.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:25.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T04:07:25.421 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:25.421 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[88407]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:25.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:25.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T04:07:25.422 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:25 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:25 smithi153 ceph-mon[67960]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi098.wixybe", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-12T04:07:25.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:25 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:25.702 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:07:25.702 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (9m) 39s ago 31m 27.4M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:07:25.702 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (9m) 40s ago 31m 89.6M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:07:25.702 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (9m) 39s ago 30m 65.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (10m) 40s ago 36m 453M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (20m) 39s ago 37m 514M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (7m) 39s ago 37m 57.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (8m) 40s ago 36m 50.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (7m) 39s ago 36m 45.7M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (10m) 39s ago 32m 14.6M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (10m) 40s ago 32m 15.2M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (6m) 39s ago 35m 53.7M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (5m) 39s ago 35m 54.2M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (4m) 39s ago 34m 56.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (3m) 39s ago 34m 59.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (3m) 40s ago 34m 52.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (2m) 40s ago 33m 48.4M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (2m) 40s ago 33m 53.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 23a1ba5d3665 2024-03-12T04:07:25.703 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (88s) 40s ago 33m 54.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 782d8be774e8 2024-03-12T04:07:25.704 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (10m) 40s ago 32m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:07:25.704 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (49s) 39s ago 30m 78.5M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 12a7e0e5eaeb 2024-03-12T04:07:25.704 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (44s) 40s ago 30m 75.2M - 19.0.0-1979-g2b0101eb 4133f8bc41bb ed96ecebb071 2024-03-12T04:07:26.097 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 8 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:07:26.098 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:07:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:07:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 15 2024-03-12T04:07:26.099 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:07:26.099 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:07:26.432 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [ 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "mon", 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr", 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "osd", 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw" 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "15/21 daemons upgraded", 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Currently upgrading iscsi daemons", 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": false 2024-03-12T04:07:26.433 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:07:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[86021]: Upgrade: Updating iscsi.foo.smithi098.wixybe 2024-03-12T04:07:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[86021]: Deploying daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T04:07:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[86021]: from='client.54774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[86021]: from='client.64711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/63112642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:26.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:26.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[88407]: Upgrade: Updating iscsi.foo.smithi098.wixybe 2024-03-12T04:07:26.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[88407]: Deploying daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T04:07:26.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[88407]: from='client.54774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:26.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[88407]: from='client.64711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:26.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/63112642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:26.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:26 smithi153 ceph-mon[67960]: Upgrade: Updating iscsi.foo.smithi098.wixybe 2024-03-12T04:07:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:26 smithi153 ceph-mon[67960]: Deploying daemon iscsi.foo.smithi098.wixybe on smithi098 2024-03-12T04:07:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:26 smithi153 ceph-mon[67960]: from='client.54774 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:26 smithi153 ceph-mon[67960]: from='client.64711 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:26 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/63112642' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:07:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:26.857 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_OK 2024-03-12T04:07:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:27 smithi153 ceph-mon[67960]: from='client.64717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:27 smithi153 ceph-mon[67960]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:27.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:27 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/485154792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:07:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:27 smithi098 ceph-mon[86021]: from='client.64717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:27 smithi098 ceph-mon[86021]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:27.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:27 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/485154792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:07:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:27 smithi098 ceph-mon[88407]: from='client.64717 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:27 smithi098 ceph-mon[88407]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:27.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:27 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/485154792' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:07:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:28 smithi153 ceph-mon[67960]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:28.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:28 smithi098 ceph-mon[86021]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:28 smithi098 ceph-mon[88407]: from='client.64729 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:29 smithi153 ceph-mon[67960]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:29.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:29 smithi153 ceph-mon[67960]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:29 smithi098 ceph-mon[88407]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:29 smithi098 ceph-mon[88407]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:29 smithi098 ceph-mon[86021]: from='client.25642 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:07:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:29 smithi098 ceph-mon[86021]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:07:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:07:30] "GET /metrics HTTP/1.1" 200 38406 "" "Prometheus/2.43.0" 2024-03-12T04:07:31.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:31 smithi153 ceph-mon[67960]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:31.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:31 smithi098 ceph-mon[86021]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:31.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:31 smithi098 ceph-mon[88407]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:33.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:33 smithi153 ceph-mon[67960]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:33.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:33 smithi098 ceph-mon[86021]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:33.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:33 smithi098 ceph-mon[88407]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:35.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:35 smithi153 ceph-mon[67960]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:35.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:35 smithi098 ceph-mon[86021]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:35.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:35 smithi098 ceph-mon[88407]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:37.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:37 smithi098 ceph-mon[86021]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:37.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:37 smithi098 ceph-mon[88407]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:37 smithi153 ceph-mon[67960]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:38.682 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:07:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:07:38.386+0000 7f799f2e6700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. 2024-03-12T04:07:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:39 smithi153 ceph-mon[67960]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:07:39.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[86021]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[88407]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:07:39.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:07:39.825 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:40.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:07:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:07:40] "GET /metrics HTTP/1.1" 200 38406 "" "Prometheus/2.43.0" 2024-03-12T04:07:40.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:40 smithi098 ceph-mon[86021]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. 2024-03-12T04:07:40.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:40 smithi098 ceph-mon[86021]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-03-12T04:07:40.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:40 smithi098 ceph-mon[86021]: Health check failed: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-12T04:07:40.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:40 smithi098 ceph-mon[88407]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. 2024-03-12T04:07:40.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:40 smithi098 ceph-mon[88407]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-03-12T04:07:40.575 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:40 smithi098 ceph-mon[88407]: Health check failed: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-12T04:07:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:40 smithi153 ceph-mon[67960]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. 2024-03-12T04:07:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:40 smithi153 ceph-mon[67960]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-03-12T04:07:40.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:40 smithi153 ceph-mon[67960]: Health check failed: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-12T04:07:42.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:42 smithi098 ceph-mon[86021]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-03-12T04:07:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:42.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:42 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:42 smithi098 ceph-mon[88407]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-03-12T04:07:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:42.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:42 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:42 smithi153 ceph-mon[67960]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1011 B/s rd, 0 op/s 2024-03-12T04:07:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:07:42.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:42 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:44.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:44 smithi098 ceph-mon[86021]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-03-12T04:07:44.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:44 smithi098 ceph-mon[88407]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-03-12T04:07:44.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:44 smithi153 ceph-mon[67960]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-03-12T04:07:46.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:46 smithi098 ceph-mon[86021]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-03-12T04:07:46.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:46 smithi098 ceph-mon[88407]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-03-12T04:07:46.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:46 smithi153 ceph-mon[67960]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 708 B/s rd, 0 op/s 2024-03-12T04:07:48.563 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:48 smithi098 ceph-mon[86021]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-03-12T04:07:48.563 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:48 smithi098 ceph-mon[88407]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-03-12T04:07:48.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:48 smithi153 ceph-mon[67960]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-03-12T04:07:50.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:07:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:07:50] "GET /metrics HTTP/1.1" 200 38483 "" "Prometheus/2.43.0" 2024-03-12T04:07:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:50 smithi098 ceph-mon[88407]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-03-12T04:07:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:50 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/822584225' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T04:07:50.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:50 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/482089825' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/999772115"}]: dispatch 2024-03-12T04:07:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:50 smithi098 ceph-mon[86021]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-03-12T04:07:50.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:50 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/822584225' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T04:07:50.575 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:50 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/482089825' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/999772115"}]: dispatch 2024-03-12T04:07:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:50 smithi153 ceph-mon[67960]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 202 B/s rd, 0 op/s 2024-03-12T04:07:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:50 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/822584225' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-12T04:07:50.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:50 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/482089825' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/999772115"}]: dispatch 2024-03-12T04:07:51.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:51 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/482089825' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/999772115"}]': finished 2024-03-12T04:07:51.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:51 smithi098 ceph-mon[86021]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T04:07:51.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:51 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2112563638' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2963866871"}]: dispatch 2024-03-12T04:07:51.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:51 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/482089825' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/999772115"}]': finished 2024-03-12T04:07:51.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:51 smithi098 ceph-mon[88407]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T04:07:51.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:51 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2112563638' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2963866871"}]: dispatch 2024-03-12T04:07:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:51 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/482089825' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/999772115"}]': finished 2024-03-12T04:07:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:51 smithi153 ceph-mon[67960]: osdmap e155: 8 total, 8 up, 8 in 2024-03-12T04:07:51.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:51 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2112563638' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2963866871"}]: dispatch 2024-03-12T04:07:52.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[86021]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-12T04:07:52.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2112563638' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2963866871"}]': finished 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[86021]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1663170709' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]: dispatch 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[86021]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]: dispatch 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[88407]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2112563638' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2963866871"}]': finished 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[88407]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1663170709' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]: dispatch 2024-03-12T04:07:52.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:52 smithi098 ceph-mon[88407]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]: dispatch 2024-03-12T04:07:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:52 smithi153 ceph-mon[67960]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2024-03-12T04:07:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:52 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2112563638' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2963866871"}]': finished 2024-03-12T04:07:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:52 smithi153 ceph-mon[67960]: osdmap e156: 8 total, 8 up, 8 in 2024-03-12T04:07:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:52 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1663170709' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]: dispatch 2024-03-12T04:07:52.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:52 smithi153 ceph-mon[67960]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]: dispatch 2024-03-12T04:07:53.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:53 smithi098 ceph-mon[86021]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]': finished 2024-03-12T04:07:53.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:53 smithi098 ceph-mon[86021]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T04:07:53.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:53 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/4011259250' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/423142338"}]: dispatch 2024-03-12T04:07:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:53 smithi098 ceph-mon[88407]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]': finished 2024-03-12T04:07:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:53 smithi098 ceph-mon[88407]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T04:07:53.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:53 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/4011259250' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/423142338"}]: dispatch 2024-03-12T04:07:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:53 smithi153 ceph-mon[67960]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/2052594852"}]': finished 2024-03-12T04:07:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:53 smithi153 ceph-mon[67960]: osdmap e157: 8 total, 8 up, 8 in 2024-03-12T04:07:53.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:53 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/4011259250' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/423142338"}]: dispatch 2024-03-12T04:07:54.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[88407]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T04:07:54.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/4011259250' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/423142338"}]': finished 2024-03-12T04:07:54.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[88407]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T04:07:54.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1872428562' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3684417667"}]: dispatch 2024-03-12T04:07:54.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[86021]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T04:07:54.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/4011259250' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/423142338"}]': finished 2024-03-12T04:07:54.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[86021]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T04:07:54.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:54 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1872428562' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3684417667"}]: dispatch 2024-03-12T04:07:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:54 smithi153 ceph-mon[67960]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-12T04:07:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:54 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/4011259250' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6801/423142338"}]': finished 2024-03-12T04:07:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:54 smithi153 ceph-mon[67960]: osdmap e158: 8 total, 8 up, 8 in 2024-03-12T04:07:54.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:54 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1872428562' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3684417667"}]: dispatch 2024-03-12T04:07:55.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1872428562' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3684417667"}]': finished 2024-03-12T04:07:55.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[86021]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T04:07:55.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/4041646102' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]: dispatch 2024-03-12T04:07:55.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[86021]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]: dispatch 2024-03-12T04:07:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1872428562' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3684417667"}]': finished 2024-03-12T04:07:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[88407]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T04:07:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/4041646102' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]: dispatch 2024-03-12T04:07:55.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:55 smithi098 ceph-mon[88407]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]: dispatch 2024-03-12T04:07:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:55 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1872428562' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:0/3684417667"}]': finished 2024-03-12T04:07:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:55 smithi153 ceph-mon[67960]: osdmap e159: 8 total, 8 up, 8 in 2024-03-12T04:07:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:55 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/4041646102' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]: dispatch 2024-03-12T04:07:55.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:55 smithi153 ceph-mon[67960]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]: dispatch 2024-03-12T04:07:56.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[86021]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:07:56.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[86021]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]': finished 2024-03-12T04:07:56.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[86021]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T04:07:56.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[88407]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:07:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[88407]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]': finished 2024-03-12T04:07:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[88407]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T04:07:56.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:56.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:56 smithi153 ceph-mon[67960]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-12T04:07:56.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:56 smithi153 ceph-mon[67960]: from='client.? ' entity='client.iscsi.foo.smithi098.wixybe' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.98:6800/423142338"}]': finished 2024-03-12T04:07:56.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:56 smithi153 ceph-mon[67960]: osdmap e160: 8 total, 8 up, 8 in 2024-03-12T04:07:56.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:07:57.217 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:07:57.545 INFO:teuthology.orchestra.run.smithi098.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed." 2024-03-12T04:07:58.243 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T04:07:58.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:58 smithi098 ceph-mon[86021]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:58.554 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:58 smithi098 ceph-mon[86021]: from='client.64801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:58.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:58 smithi098 ceph-mon[88407]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:58.555 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:58 smithi098 ceph-mon[88407]: from='client.64801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:58 smithi153 ceph-mon[67960]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:07:58.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:58 smithi153 ceph-mon[67960]: from='client.64801 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:59.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:07:59 smithi098 ceph-mon[86021]: from='client.54876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:59.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:07:59 smithi098 ceph-mon[88407]: from='client.54876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:07:59.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:07:59 smithi153 ceph-mon[67960]: from='client.54876 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (10m) 74s ago 32m 27.4M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (10m) 74s ago 31m 89.6M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (9m) 74s ago 31m 65.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (11m) 74s ago 36m 453M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (21m) 74s ago 38m 514M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (8m) 74s ago 38m 57.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (9m) 74s ago 36m 50.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (7m) 74s ago 37m 45.7M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (11m) 74s ago 32m 14.6M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:08:00.365 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (10m) 74s ago 32m 15.2M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (6m) 74s ago 36m 53.7M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (6m) 74s ago 35m 54.2M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (5m) 74s ago 35m 56.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (4m) 74s ago 35m 59.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (3m) 74s ago 34m 52.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (3m) 74s ago 34m 48.4M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (2m) 74s ago 34m 53.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 23a1ba5d3665 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (2m) 74s ago 33m 54.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 782d8be774e8 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (10m) 74s ago 33m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (84s) 74s ago 31m 78.5M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 12a7e0e5eaeb 2024-03-12T04:08:00.366 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (78s) 74s ago 31m 75.2M - 19.0.0-1979-g2b0101eb 4133f8bc41bb ed96ecebb071 2024-03-12T04:08:00.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:00 smithi098 ceph-mon[86021]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:00.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:00 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:00.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:08:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:08:00] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-03-12T04:08:00.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:00 smithi098 ceph-mon[88407]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:00.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:00 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:00.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:00 smithi153 ceph-mon[67960]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:00.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:00 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:01.017 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions' 2024-03-12T04:08:01.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:01 smithi153 ceph-mon[67960]: from='client.64813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:08:01.443 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:08:01 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a[66347]: logger=cleanup t=2024-03-12T04:08:01.161887244Z level=info msg="Completed cleanup jobs" duration=1.242269ms 2024-03-12T04:08:01.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:01 smithi098 ceph-mon[86021]: from='client.64813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:08:01.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:01 smithi098 ceph-mon[88407]: from='client.64813 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:08:02.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:02 smithi098 ceph-mon[86021]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-12T04:08:02.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:02 smithi098 ceph-mon[88407]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-12T04:08:02.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:02 smithi153 ceph-mon[67960]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 8 2024-03-12T04:08:03.061 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:08:03.062 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:08:03.062 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:08:03.062 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:08:03.062 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:08:03.062 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 15 2024-03-12T04:08:03.062 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:08:03.062 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:08:03.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:03 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/224920961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:08:03.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:03 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/224920961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:08:03.692 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-12T04:08:03.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:03 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/224920961' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:08:04.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:04 smithi098 ceph-mon[86021]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:04.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:04 smithi098 ceph-mon[88407]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:04.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:04 smithi153 ceph-mon[67960]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:05.537 INFO:teuthology.orchestra.run.smithi098.stdout:wait for servicemap items w/ changing names to refresh 2024-03-12T04:08:06.204 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'sleep 60' 2024-03-12T04:08:06.644 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:06 smithi098 ceph-mon[86021]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:08:06.645 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:06 smithi098 ceph-mon[88407]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:08:06.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:06 smithi153 ceph-mon[67960]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-12T04:08:08.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:08 smithi153 ceph-mon[67960]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T04:08:08.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:08 smithi098 ceph-mon[86021]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T04:08:08.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:08 smithi098 ceph-mon[88407]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-12T04:08:10.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:10 smithi098 ceph-mon[86021]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:10.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:10 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:10.574 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:08:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:08:10] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-03-12T04:08:10.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:10 smithi098 ceph-mon[88407]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:10.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:10 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:10.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:10 smithi153 ceph-mon[67960]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:10.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:10 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:11.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:11.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:11.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:12.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:12 smithi153 ceph-mon[67960]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:12.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:12 smithi098 ceph-mon[88407]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:12.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:12 smithi098 ceph-mon[86021]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:14.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:14 smithi153 ceph-mon[67960]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:14.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:14 smithi098 ceph-mon[86021]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:14.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:14 smithi098 ceph-mon[88407]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:16.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:16 smithi153 ceph-mon[67960]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:16.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:16 smithi098 ceph-mon[86021]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:16.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:16 smithi098 ceph-mon[88407]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:18.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:18 smithi153 ceph-mon[67960]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:18.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:18 smithi098 ceph-mon[86021]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:18.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:18 smithi098 ceph-mon[88407]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:08:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:08:20] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-03-12T04:08:20.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:20 smithi098 ceph-mon[86021]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:20.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:20 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:20.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:20 smithi098 ceph-mon[88407]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:20.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:20 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:20.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:20 smithi153 ceph-mon[67960]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:20.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:20 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:22.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:22 smithi153 ceph-mon[67960]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:22.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:22 smithi098 ceph-mon[86021]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:22.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:22 smithi098 ceph-mon[88407]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:24.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:24 smithi153 ceph-mon[67960]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:24.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:24 smithi098 ceph-mon[86021]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:24.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:24 smithi098 ceph-mon[88407]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:26 smithi153 ceph-mon[67960]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:26.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:26.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:26 smithi098 ceph-mon[86021]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:26.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:26 smithi098 ceph-mon[88407]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:26.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:28.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:28 smithi153 ceph-mon[67960]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:28.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:28 smithi098 ceph-mon[86021]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:28.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:28 smithi098 ceph-mon[88407]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:30.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:08:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:08:30] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-03-12T04:08:30.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:30 smithi098 ceph-mon[88407]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:30.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:30 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:30 smithi098 ceph-mon[86021]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:30.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:30 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:30 smithi153 ceph-mon[67960]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:30.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:30 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:32.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:32 smithi153 ceph-mon[67960]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:32.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:32 smithi098 ceph-mon[88407]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:32.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:32 smithi098 ceph-mon[86021]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:34.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:34 smithi153 ceph-mon[67960]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:34.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:34 smithi098 ceph-mon[88407]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:34.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:34 smithi098 ceph-mon[86021]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:36.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:36 smithi153 ceph-mon[67960]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:36.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:36 smithi098 ceph-mon[88407]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:36.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:36 smithi098 ceph-mon[86021]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:37 smithi153 ceph-mon[67960]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:37.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:37 smithi098 ceph-mon[86021]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:37.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:37 smithi098 ceph-mon[88407]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:39.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[88407]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:39.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[86021]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:08:39.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:39 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:08:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:39 smithi153 ceph-mon[67960]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:39 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-12T04:08:39.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-12T04:08:39.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:39 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' 2024-03-12T04:08:40.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:08:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:08:40] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-03-12T04:08:41.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:41 smithi098 ceph-mon[86021]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:41.726 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:41 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:41.726 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:41 smithi098 ceph-mon[88407]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:41.727 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:41 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:41 smithi153 ceph-mon[67960]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:41.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:41 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:43.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:43 smithi098 ceph-mon[86021]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:43.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:43 smithi098 ceph-mon[88407]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:43.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:43 smithi153 ceph-mon[67960]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:45.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:45 smithi098 ceph-mon[86021]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:45.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:45 smithi098 ceph-mon[88407]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:45.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:45 smithi153 ceph-mon[67960]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:47.744 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:47 smithi153 ceph-mon[67960]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:47.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:47 smithi098 ceph-mon[86021]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:47.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:47 smithi098 ceph-mon[88407]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:49.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:49 smithi098 ceph-mon[88407]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:49.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:49 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:49.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:49 smithi098 ceph-mon[86021]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:49.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:49 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:49.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:49 smithi153 ceph-mon[67960]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:49.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:49 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:50.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:08:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:08:50] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-03-12T04:08:51.796 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:51 smithi098 ceph-mon[86021]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:51.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:51 smithi098 ceph-mon[88407]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:51.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:51 smithi153 ceph-mon[67960]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:53.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:53 smithi098 ceph-mon[86021]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:53.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:53 smithi098 ceph-mon[88407]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:53.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:53 smithi153 ceph-mon[67960]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:55.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:55 smithi098 ceph-mon[86021]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:55.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:55 smithi098 ceph-mon[88407]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:55.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:55 smithi153 ceph-mon[67960]: pgmap v320: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:08:56.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:56 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:56.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:56 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:56.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:56 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:08:57.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:57 smithi098 ceph-mon[86021]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:57.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:57 smithi098 ceph-mon[88407]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:57.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:57 smithi153 ceph-mon[67960]: pgmap v321: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:08:59.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:59 smithi098 ceph-mon[86021]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:59.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:08:59 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:59 smithi098 ceph-mon[88407]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:59.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:08:59 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:08:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:59 smithi153 ceph-mon[67960]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-12T04:08:59.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:08:59 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:00.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:00 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:09:00] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-03-12T04:09:01.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:01 smithi098 ceph-mon[86021]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:01.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:01 smithi098 ceph-mon[88407]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:01.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:01 smithi153 ceph-mon[67960]: pgmap v323: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:03.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:03 smithi098 ceph-mon[86021]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:03.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:03 smithi098 ceph-mon[88407]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:03.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:03 smithi153 ceph-mon[67960]: pgmap v324: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:05.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:05 smithi098 ceph-mon[86021]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:05.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:05 smithi098 ceph-mon[88407]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:05.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:05 smithi153 ceph-mon[67960]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:07.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:07 smithi098 ceph-mon[86021]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:07.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:07 smithi098 ceph-mon[88407]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:07.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:07 smithi153 ceph-mon[67960]: pgmap v326: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:08.709 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ps' 2024-03-12T04:09:09.702 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:09 smithi098 ceph-mon[86021]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:09.702 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:09 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:09.703 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:09 smithi098 ceph-mon[88407]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:09.703 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:09 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:09.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:09 smithi153 ceph-mon[67960]: pgmap v327: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:09.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:09 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:10.274 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:10 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:09:10] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-03-12T04:09:10.709 INFO:teuthology.orchestra.run.smithi098.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-12T04:09:10.709 INFO:teuthology.orchestra.run.smithi098.stdout:alertmanager.a smithi098 *:9093,9094 running (11m) 2m ago 33m 27.4M - 0.25.0 c8568f914cd2 5fd1b9f8e512 2024-03-12T04:09:10.709 INFO:teuthology.orchestra.run.smithi098.stdout:grafana.a smithi153 *:3000 running (11m) 2m ago 33m 89.6M - 9.4.12 a0ea5e3c46d9 499a74c577ce 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:iscsi.foo.smithi098.wixybe smithi098 running (10m) 2m ago 32m 65.0M - 3.5 e1d6a67b021e de0d7e2dd549 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.x smithi153 *:8443,9283,8765 running (12m) 2m ago 37m 453M - 19.0.0-1979-g2b0101eb 4133f8bc41bb bc9c9a5eb6c2 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:mgr.y smithi098 *:8443,9283,8765 running (22m) 2m ago 39m 514M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 20b96dbb429c 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:mon.a smithi098 running (9m) 2m ago 39m 57.2M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb ee9742767c62 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:mon.b smithi153 running (10m) 2m ago 38m 50.0M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 17f05af62f31 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:mon.c smithi098 running (8m) 2m ago 38m 45.7M 2048M 19.0.0-1979-g2b0101eb 4133f8bc41bb 3a3af35df14b 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.a smithi098 *:9100 running (12m) 2m ago 34m 14.6M - 1.5.0 0da6a335fe13 ba4b1fd77bbb 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:node-exporter.b smithi153 *:9100 running (12m) 2m ago 34m 15.2M - 1.5.0 0da6a335fe13 a0fe78879ec5 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:osd.0 smithi098 running (7m) 2m ago 37m 53.7M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb a109e6088552 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:osd.1 smithi098 running (7m) 2m ago 37m 54.2M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb 4c06348cf15a 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:osd.2 smithi098 running (6m) 2m ago 36m 56.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb f80bdb9385b3 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:osd.3 smithi098 running (5m) 2m ago 36m 59.5M 2992M 19.0.0-1979-g2b0101eb 4133f8bc41bb b8b05419735b 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:osd.4 smithi153 running (4m) 2m ago 35m 52.0M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb e913f3621dd5 2024-03-12T04:09:10.710 INFO:teuthology.orchestra.run.smithi098.stdout:osd.5 smithi153 running (4m) 2m ago 35m 48.4M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 42f3b944dd81 2024-03-12T04:09:10.711 INFO:teuthology.orchestra.run.smithi098.stdout:osd.6 smithi153 running (3m) 2m ago 35m 53.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 23a1ba5d3665 2024-03-12T04:09:10.711 INFO:teuthology.orchestra.run.smithi098.stdout:osd.7 smithi153 running (3m) 2m ago 34m 54.1M 3248M 19.0.0-1979-g2b0101eb 4133f8bc41bb 782d8be774e8 2024-03-12T04:09:10.711 INFO:teuthology.orchestra.run.smithi098.stdout:prometheus.a smithi153 *:9095 running (11m) 2m ago 34m 76.7M - 2.43.0 a07b618ecd1d 7597e44b0bb5 2024-03-12T04:09:10.711 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi098.tgzbnw smithi098 *:8000 running (2m) 2m ago 32m 78.5M - 19.0.0-1979-g2b0101eb 4133f8bc41bb 12a7e0e5eaeb 2024-03-12T04:09:10.711 INFO:teuthology.orchestra.run.smithi098.stdout:rgw.foo.smithi153.tuboyg smithi153 *:8000 running (2m) 2m ago 32m 75.2M - 19.0.0-1979-g2b0101eb 4133f8bc41bb ed96ecebb071 2024-03-12T04:09:11.413 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions' 2024-03-12T04:09:11.665 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:11 smithi098 ceph-mon[88407]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:11.665 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:11 smithi098 ceph-mon[88407]: from='client.64825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:11.666 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:11 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:09:11.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:11 smithi098 ceph-mon[86021]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:11.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:11 smithi098 ceph-mon[86021]: from='client.64825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:11.666 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:11 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:09:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:11 smithi153 ceph-mon[67960]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:11 smithi153 ceph-mon[67960]: from='client.64825 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:11.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:11 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:09:13.431 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:09:13.431 INFO:teuthology.orchestra.run.smithi098.stdout: "mon": { 2024-03-12T04:09:13.431 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 3 2024-03-12T04:09:13.431 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:09:13.431 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr": { 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: "osd": { 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 8 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw": { 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 2 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: }, 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: "overall": { 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)": 15 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout: } 2024-03-12T04:09:13.432 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:09:13.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:13 smithi098 ceph-mon[86021]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:13.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:13 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2083674704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:13.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:13 smithi098 ceph-mon[88407]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:13.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:13 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2083674704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:13 smithi153 ceph-mon[67960]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:13.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:13 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2083674704' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:14.043 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch upgrade status' 2024-03-12T04:09:15.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:15 smithi098 ceph-mon[86021]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:15.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:15 smithi098 ceph-mon[88407]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:15.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:15 smithi153 ceph-mon[67960]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b", 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "in_progress": true, 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "services_complete": [ 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "mon", 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "mgr", 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "osd", 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: "rgw" 2024-03-12T04:09:15.990 INFO:teuthology.orchestra.run.smithi098.stdout: ], 2024-03-12T04:09:15.991 INFO:teuthology.orchestra.run.smithi098.stdout: "progress": "15/21 daemons upgraded", 2024-03-12T04:09:15.991 INFO:teuthology.orchestra.run.smithi098.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed.", 2024-03-12T04:09:15.991 INFO:teuthology.orchestra.run.smithi098.stdout: "is_paused": true 2024-03-12T04:09:15.991 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:09:16.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:16 smithi098 ceph-mon[86021]: from='client.64837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:16.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:16 smithi098 ceph-mon[88407]: from='client.64837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:16.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph health detail' 2024-03-12T04:09:16.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:16 smithi153 ceph-mon[67960]: from='client.64837 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:17.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:17 smithi098 ceph-mon[88407]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:17.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:17 smithi098 ceph-mon[86021]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:17.843 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:17 smithi153 ceph-mon[67960]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:18.685 INFO:teuthology.orchestra.run.smithi098.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi098.wixybe on host smithi098 failed. 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout: Upgrade daemon: iscsi.foo.smithi098.wixybe: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi098.wixybe ... 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:Creating ceph-iscsi config... 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/iscsi-gateway.cfg 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:Write file: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/iscsi.foo.smithi098.wixybe/tcmu-runner-entrypoint.sh 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2dac0aaff8\x2de020\x2d11ee\x2d95c8\x2d87774f69a715.slice/ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:Non-zero exit code 1 from systemctl start ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:systemctl: stderr See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T04:09:18.686 INFO:teuthology.orchestra.run.smithi098.stdout:Traceback (most recent call last): 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: return _run_code(code, main_globals, None, 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: exec(code, run_globals) 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5527, in 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 5515, in main 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3043, in command_deploy_from 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3078, in _common_deploy 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 3098, in _deploy_daemon_container 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1034, in deploy_daemon 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/__main__.py", line 1170, in deploy_daemon_units 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout: File "/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/cephadm.bd8d5b863d617ca072394bbcbe4d2c7b6bc99d9ff96b3f2cec1c64334f7433e7/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout:RuntimeError: Failed command: systemctl start ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe: Job for ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service failed because the control process exited with error code. 2024-03-12T04:09:18.687 INFO:teuthology.orchestra.run.smithi098.stdout:See "systemctl status ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" and "journalctl -xeu ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@iscsi.foo.smithi098.wixybe.service" for details. 2024-03-12T04:09:19.324 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-12T04:09:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:19 smithi098 ceph-mon[86021]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:19.573 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:19 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/952982786' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:09:19.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:19 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:19 smithi098 ceph-mon[88407]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:19 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/952982786' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:09:19.574 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:19 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:19 smithi153 ceph-mon[67960]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:19 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/952982786' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-12T04:09:19.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:19 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:20.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:20 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:09:20] "GET /metrics HTTP/1.1" 200 38487 "" "Prometheus/2.43.0" 2024-03-12T04:09:21.397 INFO:teuthology.orchestra.run.smithi098.stdout:true 2024-03-12T04:09:21.689 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:21 smithi098 ceph-mon[86021]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:21.689 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:21 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/1717757599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:21.689 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:21 smithi098 ceph-mon[88407]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:21.689 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:21 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/1717757599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:21 smithi153 ceph-mon[67960]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:21.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:21 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/1717757599' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:21.971 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-12T04:09:23.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:23 smithi098 ceph-mon[86021]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:23.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:23 smithi098 ceph-mon[88407]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:23.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:23 smithi153 ceph-mon[67960]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:24.026 INFO:teuthology.orchestra.run.smithi098.stdout: "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)" 2024-03-12T04:09:24.605 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -e sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-12T04:09:24.631 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:24 smithi098 ceph-mon[86021]: from='client.? 172.21.15.98:0/2969432860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:24.631 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:24 smithi098 ceph-mon[88407]: from='client.? 172.21.15.98:0/2969432860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:24.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:24 smithi153 ceph-mon[67960]: from='client.? 172.21.15.98:0/2969432860' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-12T04:09:25.780 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:25 smithi098 ceph-mon[86021]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:25.780 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:25 smithi098 ceph-mon[88407]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:25.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:25 smithi153 ceph-mon[67960]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:26.501 INFO:teuthology.orchestra.run.smithi098.stdout:osd 8 2m ago - 2024-03-12T04:09:26.795 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:26 smithi098 ceph-mon[86021]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:09:26.796 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:26 smithi098 ceph-mon[88407]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:09:26.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:26 smithi153 ceph-mon[67960]: from='mgr.25819 172.21.15.98:0/1755382421' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-12T04:09:27.103 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-12T04:09:27.114 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi098.front.sepia.ceph.com 2024-03-12T04:09:27.114 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-12T04:09:27.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:27 smithi098 ceph-mon[86021]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:27.784 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:27 smithi098 ceph-mon[86021]: from='client.64861 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:27.784 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:27 smithi098 ceph-mon[88407]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:27.785 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:27 smithi098 ceph-mon[88407]: from='client.64861 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:27 smithi153 ceph-mon[67960]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:27.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:27 smithi153 ceph-mon[67960]: from='client.64861 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:29 smithi098 ceph-mon[88407]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:29 smithi098 ceph-mon[88407]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:29.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:29 smithi098 ceph-mon[88407]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:29 smithi098 ceph-mon[86021]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:29 smithi098 ceph-mon[86021]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:29.824 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:29 smithi098 ceph-mon[86021]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:29.836 INFO:teuthology.orchestra.run.smithi098.stdout:{ 2024-03-12T04:09:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "image": "quay.io/ceph/ceph", 2024-03-12T04:09:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "registry": "quay.io", 2024-03-12T04:09:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "bare_image": "ceph/ceph", 2024-03-12T04:09:29.836 INFO:teuthology.orchestra.run.smithi098.stdout: "versions": [] 2024-03-12T04:09:29.836 INFO:teuthology.orchestra.run.smithi098.stdout:} 2024-03-12T04:09:29.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:29 smithi153 ceph-mon[67960]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:29.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:29 smithi153 ceph-mon[67960]: from='client.54936 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:29.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:29 smithi153 ceph-mon[67960]: from='client.54819 -' entity='client.iscsi.foo.smithi098.wixybe' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-12T04:09:30.455 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:30 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:09:30] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-03-12T04:09:30.479 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-12T04:09:31.746 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:31 smithi098 ceph-mon[86021]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:31.746 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:31 smithi098 ceph-mon[88407]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:31.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:31 smithi153 ceph-mon[67960]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:33.189 INFO:teuthology.orchestra.run.smithi098.stdout: "16.2.0", 2024-03-12T04:09:33.812 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:33 smithi098 ceph-mon[86021]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:33.812 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:33 smithi098 ceph-mon[86021]: from='client.54942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:33.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:33 smithi098 ceph-mon[88407]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:33.813 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:33 smithi098 ceph-mon[88407]: from='client.54942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:33.837 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-12T04:09:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:33 smithi153 ceph-mon[67960]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:33.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:33 smithi153 ceph-mon[67960]: from='client.54942 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:35.823 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:35 smithi098 ceph-mon[86021]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:35.824 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:35 smithi098 ceph-mon[88407]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:35.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:35 smithi153 ceph-mon[67960]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-12T04:09:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "v16.2.2", 2024-03-12T04:09:36.453 INFO:teuthology.orchestra.run.smithi098.stdout: "v16.2.2-20210505", 2024-03-12T04:09:36.819 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:36 smithi098 ceph-mon[86021]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:36.819 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:36 smithi098 ceph-mon[88407]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:36.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:36 smithi153 ceph-mon[67960]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-12T04:09:37.282 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-12T04:09:37.292 INFO:tasks.cephadm:Teardown begin 2024-03-12T04:09:37.292 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T04:09:37.325 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T04:09:37.361 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-12T04:09:37.361 DEBUG:teuthology.orchestra.run.smithi098:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T04:09:37.383 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-12T04:09:37.418 INFO:tasks.cephadm:Stopping all daemons... 2024-03-12T04:09:37.419 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-12T04:09:37.419 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a 2024-03-12T04:09:37.574 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:37 smithi098 systemd[1]: Stopping Ceph mon.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:09:37.692 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:37 smithi153 ceph-mon[67960]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:37.848 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-mon[88407]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:37.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-mon[86021]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 262 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-12T04:09:37.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a[86017]: 2024-03-12T04:09:37.720+0000 7f2b8f394700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:09:37.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a[86017]: 2024-03-12T04:09:37.721+0000 7f2b8f394700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-12T04:09:37.849 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:37 smithi098 podman[129722]: 2024-03-12 04:09:37.752181969 +0000 UTC m=+0.135384605 container died ee9742767c6257d6d6969b013cf4f0b9bf026630889ad1143d35396980568408 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T04:09:37.849 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:09:37] ENGINE Bus STOPPING 2024-03-12T04:09:37.849 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:09:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-12T04:09:37.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:09:37] ENGINE Bus STOPPED 2024-03-12T04:09:37.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:09:37] ENGINE Bus STARTING 2024-03-12T04:09:37.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: CherryPy Checker: 2024-03-12T04:09:37.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: The Application mounted at '' has an empty config. 2024-03-12T04:09:37.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:09:37.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:09:37] ENGINE Serving on http://:::9283 2024-03-12T04:09:37.850 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:37 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: [12/Mar/2024:04:09:37] ENGINE Bus STARTED 2024-03-12T04:09:38.467 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:38 smithi098 podman[129722]: 2024-03-12 04:09:38.13850868 +0000 UTC m=+0.521711323 container cleanup ee9742767c6257d6d6969b013cf4f0b9bf026630889ad1143d35396980568408 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:09:38.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:38 smithi098 bash[129722]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a 2024-03-12T04:09:38.468 INFO:journalctl@ceph.mon.a.smithi098.stdout:Mar 12 04:09:38 smithi098 podman[129736]: 2024-03-12 04:09:38.356559047 +0000 UTC m=+0.606640540 container remove ee9742767c6257d6d6969b013cf4f0b9bf026630889ad1143d35396980568408 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-12T04:09:38.487 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.a.service' 2024-03-12T04:09:38.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:38.534 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-12T04:09:38.534 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-12T04:09:38.535 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.c 2024-03-12T04:09:38.823 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:38 smithi098 systemd[1]: Stopping Ceph mon.c for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:09:39.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c[88401]: 2024-03-12T04:09:38.844+0000 7f9b37736700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:09:39.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:38 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c[88401]: 2024-03-12T04:09:38.844+0000 7f9b37736700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-12T04:09:39.137 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:38 smithi098 podman[129854]: 2024-03-12 04:09:38.947781513 +0000 UTC m=+0.234731930 container died 3a3af35df14b6e7b117bddafc1582095543bf312c07e81ee63e46942ad8570cc (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T04:09:39.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:39 smithi098 podman[129854]: 2024-03-12 04:09:39.359823056 +0000 UTC m=+0.646773472 container cleanup 3a3af35df14b6e7b117bddafc1582095543bf312c07e81ee63e46942ad8570cc (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-12T04:09:39.573 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:39 smithi098 bash[129854]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c 2024-03-12T04:09:39.731 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.c.service' 2024-03-12T04:09:39.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:39 smithi098 podman[129868]: 2024-03-12 04:09:39.594830825 +0000 UTC m=+0.640432668 container remove 3a3af35df14b6e7b117bddafc1582095543bf312c07e81ee63e46942ad8570cc (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-12T04:09:39.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:39 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-12T04:09:39.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:39 smithi098 systemd[1]: Stopped Ceph mon.c for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:09:39.732 INFO:journalctl@ceph.mon.c.smithi098.stdout:Mar 12 04:09:39 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.c.service: Consumed 5.823s CPU time. 2024-03-12T04:09:39.768 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:39.768 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-12T04:09:39.769 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-12T04:09:39.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.b 2024-03-12T04:09:40.103 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:39 smithi153 systemd[1]: Stopping Ceph mon.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:09:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b[67956]: 2024-03-12T04:09:40.100+0000 7f0be43ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:09:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b[67956]: 2024-03-12T04:09:40.100+0000 7f0be43ca700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-12T04:09:40.442 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:40 smithi153 podman[101225]: 2024-03-12 04:09:40.335835867 +0000 UTC m=+0.368073258 container died 17f05af62f31ea6f9f2218af7d2016d50cea013bf004a7a727bd538ad1e1b6f9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-12T04:09:40.573 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:40 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: ::ffff:172.21.15.153 - - [12/Mar/2024:04:09:40] "GET /metrics HTTP/1.1" 200 38489 "" "Prometheus/2.43.0" 2024-03-12T04:09:40.942 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:40 smithi153 podman[101225]: 2024-03-12 04:09:40.785109729 +0000 UTC m=+0.817347119 container cleanup 17f05af62f31ea6f9f2218af7d2016d50cea013bf004a7a727bd538ad1e1b6f9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-12T04:09:40.943 INFO:journalctl@ceph.mon.b.smithi153.stdout:Mar 12 04:09:40 smithi153 bash[101225]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mon-b 2024-03-12T04:09:41.199 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mon.b.service' 2024-03-12T04:09:41.243 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:41.243 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-12T04:09:41.243 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-12T04:09:41.244 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y 2024-03-12T04:09:41.343 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:41 smithi098 systemd[1]: Stopping Ceph mgr.y for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:09:41.597 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:09:41.563+0000 7f7a0d375700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:09:41.598 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:41 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y[63024]: 2024-03-12T04:09:41.563+0000 7f7a0d375700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-12T04:09:41.892 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:41 smithi098 podman[129980]: 2024-03-12 04:09:41.606315909 +0000 UTC m=+0.160716779 container died 20b96dbb429c2dc568d5fdd15316472120f89ddfb1ba5d4ac4b27ea80ed73cc1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-2b0101e) 2024-03-12T04:09:42.242 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:41 smithi098 podman[129980]: 2024-03-12 04:09:41.964618131 +0000 UTC m=+0.519018992 container cleanup 20b96dbb429c2dc568d5fdd15316472120f89ddfb1ba5d4ac4b27ea80ed73cc1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:09:42.243 INFO:journalctl@ceph.mgr.y.smithi098.stdout:Mar 12 04:09:41 smithi098 bash[129980]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-y 2024-03-12T04:09:42.410 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-12T04:09:42.449 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:42.450 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-12T04:09:42.450 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-12T04:09:42.450 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x 2024-03-12T04:09:42.930 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 04:09:42 smithi153 systemd[1]: Stopping Ceph mgr.x for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:09:43.192 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 04:09:42 smithi153 podman[101347]: 2024-03-12 04:09:42.92882813 +0000 UTC m=+0.294112601 container died bc9c9a5eb6c22d1f07a711acc18f4e6778961745152aa11c068f2c7ec60e619e (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True) 2024-03-12T04:09:43.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 04:09:43 smithi153 podman[101347]: 2024-03-12 04:09:43.364646475 +0000 UTC m=+0.729930955 container cleanup bc9c9a5eb6c22d1f07a711acc18f4e6778961745152aa11c068f2c7ec60e619e (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-12T04:09:43.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 04:09:43 smithi153 bash[101347]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x 2024-03-12T04:09:43.692 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 04:09:43 smithi153 podman[101361]: 2024-03-12 04:09:43.558015184 +0000 UTC m=+0.630431721 container remove bc9c9a5eb6c22d1f07a711acc18f4e6778961745152aa11c068f2c7ec60e619e (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e) 2024-03-12T04:09:43.693 INFO:journalctl@ceph.mgr.x.smithi153.stdout:Mar 12 04:09:43 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-12T04:09:43.773 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-12T04:09:43.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:43.809 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-12T04:09:43.809 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-12T04:09:43.809 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.0 2024-03-12T04:09:44.095 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:43 smithi098 systemd[1]: Stopping Ceph osd.0 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:09:44.573 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[95554]: 2024-03-12T04:09:44.092+0000 7f45f8c75700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:09:44.573 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[95554]: 2024-03-12T04:09:44.092+0000 7f45f8c75700 -1 osd.0 160 *** Got signal Terminated *** 2024-03-12T04:09:44.573 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:44 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0[95554]: 2024-03-12T04:09:44.092+0000 7f45f8c75700 -1 osd.0 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:09:49.814 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:49 smithi098 podman[130105]: 2024-03-12 04:09:49.515174135 +0000 UTC m=+5.515707954 container died a109e6088552df25af19a200f56aa63309f8047084104f49df3587286e30ec00 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T04:09:50.073 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:49 smithi098 podman[130105]: 2024-03-12 04:09:49.902705368 +0000 UTC m=+5.903239180 container cleanup a109e6088552df25af19a200f56aa63309f8047084104f49df3587286e30ec00 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-12T04:09:50.073 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:49 smithi098 bash[130105]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0 2024-03-12T04:09:50.523 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:50 smithi098 podman[130122]: 2024-03-12 04:09:50.162354867 +0000 UTC m=+0.636524032 container remove a109e6088552df25af19a200f56aa63309f8047084104f49df3587286e30ec00 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T04:09:50.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:50 smithi098 podman[130210]: 2024-03-12 04:09:50.425012002 +0000 UTC m=+0.017568069 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:09:50.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:50 smithi098 podman[130210]: 2024-03-12 04:09:50.529894155 +0000 UTC m=+0.122450257 container create 513fdeae0d276d477133997b7f7c98445265aae2bc48a835d75de30e1c038879 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-03-12T04:09:50.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:50 smithi098 podman[130210]: 2024-03-12 04:09:50.76636007 +0000 UTC m=+0.358916130 container init 513fdeae0d276d477133997b7f7c98445265aae2bc48a835d75de30e1c038879 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-12T04:09:50.824 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:50 smithi098 podman[130210]: 2024-03-12 04:09:50.769659718 +0000 UTC m=+0.362215781 container start 513fdeae0d276d477133997b7f7c98445265aae2bc48a835d75de30e1c038879 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-12T04:09:51.076 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:50 smithi098 podman[130210]: 2024-03-12 04:09:50.8743348 +0000 UTC m=+0.466890869 container attach 513fdeae0d276d477133997b7f7c98445265aae2bc48a835d75de30e1c038879 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-12T04:09:51.077 INFO:journalctl@ceph.osd.0.smithi098.stdout:Mar 12 04:09:51 smithi098 podman[130210]: 2024-03-12 04:09:51.057989064 +0000 UTC m=+0.650545116 container died 513fdeae0d276d477133997b7f7c98445265aae2bc48a835d75de30e1c038879 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-12T04:09:51.651 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.0.service' 2024-03-12T04:09:51.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:51.698 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-12T04:09:51.699 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-12T04:09:51.699 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.1 2024-03-12T04:09:52.042 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:51 smithi098 systemd[1]: Stopping Ceph osd.1 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:09:52.323 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:52 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[100398]: 2024-03-12T04:09:52.039+0000 7f9d326b3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:09:52.323 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:52 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[100398]: 2024-03-12T04:09:52.039+0000 7f9d326b3700 -1 osd.1 160 *** Got signal Terminated *** 2024-03-12T04:09:52.323 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:52 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1[100398]: 2024-03-12T04:09:52.039+0000 7f9d326b3700 -1 osd.1 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:09:57.858 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:57 smithi098 podman[130317]: 2024-03-12 04:09:57.573068124 +0000 UTC m=+5.661006334 container died 4c06348cf15a649c115f22eb9223ca1f17958071dd14f90e782003fcfdb1cce1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-12T04:09:58.192 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:57 smithi098 podman[130317]: 2024-03-12 04:09:57.956217708 +0000 UTC m=+6.044155915 container cleanup 4c06348cf15a649c115f22eb9223ca1f17958071dd14f90e782003fcfdb1cce1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-12T04:09:58.192 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:57 smithi098 bash[130317]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1 2024-03-12T04:09:58.518 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:58 smithi098 podman[130335]: 2024-03-12 04:09:58.190868349 +0000 UTC m=+0.614913238 container remove 4c06348cf15a649c115f22eb9223ca1f17958071dd14f90e782003fcfdb1cce1 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True) 2024-03-12T04:09:58.779 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:58 smithi098 podman[130429]: 2024-03-12 04:09:58.516749309 +0000 UTC m=+0.109280982 container create c5cfbfd3028931b58c5f13bc93c55e75502b55429e910e9f833d5febefd49674 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T04:09:58.779 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:58 smithi098 podman[130429]: 2024-03-12 04:09:58.425587704 +0000 UTC m=+0.018119338 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:09:58.779 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:58 smithi098 podman[130429]: 2024-03-12 04:09:58.702943225 +0000 UTC m=+0.295474855 container init c5cfbfd3028931b58c5f13bc93c55e75502b55429e910e9f833d5febefd49674 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-12T04:09:58.779 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:58 smithi098 podman[130429]: 2024-03-12 04:09:58.706381836 +0000 UTC m=+0.298913468 container start c5cfbfd3028931b58c5f13bc93c55e75502b55429e910e9f833d5febefd49674 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-12T04:09:58.779 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:58 smithi098 podman[130429]: 2024-03-12 04:09:58.777826139 +0000 UTC m=+0.370357781 container attach c5cfbfd3028931b58c5f13bc93c55e75502b55429e910e9f833d5febefd49674 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-12T04:09:59.073 INFO:journalctl@ceph.osd.1.smithi098.stdout:Mar 12 04:09:58 smithi098 podman[130429]: 2024-03-12 04:09:58.987931908 +0000 UTC m=+0.580463539 container died c5cfbfd3028931b58c5f13bc93c55e75502b55429e910e9f833d5febefd49674 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T04:09:59.625 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.1.service' 2024-03-12T04:09:59.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:09:59.668 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-12T04:09:59.668 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-12T04:09:59.668 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.2 2024-03-12T04:09:59.986 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:09:59 smithi098 systemd[1]: Stopping Ceph osd.2 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:00.323 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:09:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[106987]: 2024-03-12T04:09:59.984+0000 7f33efbf6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:10:00.323 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:09:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[106987]: 2024-03-12T04:09:59.984+0000 7f33efbf6700 -1 osd.2 160 *** Got signal Terminated *** 2024-03-12T04:10:00.324 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:09:59 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2[106987]: 2024-03-12T04:09:59.984+0000 7f33efbf6700 -1 osd.2 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:10:05.771 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:05 smithi098 podman[130534]: 2024-03-12 04:10:05.515332601 +0000 UTC m=+5.631935085 container died f80bdb9385b3839071b40683bfaff4638da21112e82d38293f846a70b4c677d9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux ) 2024-03-12T04:10:06.073 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:05 smithi098 podman[130534]: 2024-03-12 04:10:05.851419372 +0000 UTC m=+5.968021859 container cleanup f80bdb9385b3839071b40683bfaff4638da21112e82d38293f846a70b4c677d9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-12T04:10:06.073 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:05 smithi098 bash[130534]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2 2024-03-12T04:10:06.073 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:06 smithi098 podman[130554]: 2024-03-12 04:10:06.019325307 +0000 UTC m=+0.502862845 container remove f80bdb9385b3839071b40683bfaff4638da21112e82d38293f846a70b4c677d9 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-12T04:10:06.724 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:06 smithi098 podman[130649]: 2024-03-12 04:10:06.334468146 +0000 UTC m=+0.017807552 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:10:06.724 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:06 smithi098 podman[130649]: 2024-03-12 04:10:06.436934858 +0000 UTC m=+0.120274302 container create 009f0c1d65c693a91be0d1e864c60db0d3faa0cdd9719b10ae59ec18eea79ddb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e) 2024-03-12T04:10:06.724 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:06 smithi098 podman[130649]: 2024-03-12 04:10:06.647741234 +0000 UTC m=+0.331080647 container init 009f0c1d65c693a91be0d1e864c60db0d3faa0cdd9719b10ae59ec18eea79ddb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-12T04:10:06.725 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:06 smithi098 podman[130649]: 2024-03-12 04:10:06.651103349 +0000 UTC m=+0.334442751 container start 009f0c1d65c693a91be0d1e864c60db0d3faa0cdd9719b10ae59ec18eea79ddb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-12T04:10:06.725 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:06 smithi098 podman[130649]: 2024-03-12 04:10:06.722782843 +0000 UTC m=+0.406122260 container attach 009f0c1d65c693a91be0d1e864c60db0d3faa0cdd9719b10ae59ec18eea79ddb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-12T04:10:07.073 INFO:journalctl@ceph.osd.2.smithi098.stdout:Mar 12 04:10:06 smithi098 podman[130649]: 2024-03-12 04:10:06.937835677 +0000 UTC m=+0.621175078 container died 009f0c1d65c693a91be0d1e864c60db0d3faa0cdd9719b10ae59ec18eea79ddb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-12T04:10:07.510 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.2.service' 2024-03-12T04:10:07.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:10:07.557 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-12T04:10:07.557 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-12T04:10:07.558 DEBUG:teuthology.orchestra.run.smithi098:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.3 2024-03-12T04:10:07.823 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:07 smithi098 systemd[1]: Stopping Ceph osd.3 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:08.323 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:07 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[113482]: 2024-03-12T04:10:07.854+0000 7f537e9c1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:10:08.323 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:07 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[113482]: 2024-03-12T04:10:07.854+0000 7f537e9c1700 -1 osd.3 160 *** Got signal Terminated *** 2024-03-12T04:10:08.323 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:07 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3[113482]: 2024-03-12T04:10:07.854+0000 7f537e9c1700 -1 osd.3 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:10:12.443 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:12.026+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:13.440 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:13 smithi098 podman[130755]: 2024-03-12 04:10:13.175848376 +0000 UTC m=+5.435708057 container died b8b05419735b77dbe9fedd26503139cdf6ee3e4157e772d9f515309054ed15cb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-12T04:10:13.442 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:12 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:12.995+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:13.807 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:13 smithi098 podman[130755]: 2024-03-12 04:10:13.537585677 +0000 UTC m=+5.797445357 container cleanup b8b05419735b77dbe9fedd26503139cdf6ee3e4157e772d9f515309054ed15cb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T04:10:13.807 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:13 smithi098 bash[130755]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3 2024-03-12T04:10:13.807 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:13.553Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T04:10:13.807 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:13 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:13.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T04:10:14.073 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:13 smithi098 podman[130772]: 2024-03-12 04:10:13.805593371 +0000 UTC m=+0.625055119 container remove b8b05419735b77dbe9fedd26503139cdf6ee3e4157e772d9f515309054ed15cb (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:14.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:13 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:13.949+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:14.193 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:13 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:13.847+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:14.411 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:14 smithi098 podman[130860]: 2024-03-12 04:10:14.059174513 +0000 UTC m=+0.019583066 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:10:14.412 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:14 smithi098 podman[130860]: 2024-03-12 04:10:14.173151783 +0000 UTC m=+0.133560333 container create c9df55f4cd62248e7a6cadc7631f9a0890c25c6f610934192e8153b5a4b38232 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-12T04:10:14.412 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:14 smithi098 podman[130860]: 2024-03-12 04:10:14.409233406 +0000 UTC m=+0.369641948 container init c9df55f4cd62248e7a6cadc7631f9a0890c25c6f610934192e8153b5a4b38232 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.build-date=20240307) 2024-03-12T04:10:14.702 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:14 smithi098 podman[130860]: 2024-03-12 04:10:14.413491523 +0000 UTC m=+0.373900071 container start c9df55f4cd62248e7a6cadc7631f9a0890c25c6f610934192e8153b5a4b38232 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e) 2024-03-12T04:10:14.702 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:14 smithi098 podman[130860]: 2024-03-12 04:10:14.492585531 +0000 UTC m=+0.452994076 container attach c9df55f4cd62248e7a6cadc7631f9a0890c25c6f610934192e8153b5a4b38232 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-12T04:10:14.942 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:14.637+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:48.309270+0000 front 2024-03-12T04:09:48.309213+0000 (oldest deadline 2024-03-12T04:10:14.209114+0000) 2024-03-12T04:10:14.943 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:14.845+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:15.047 INFO:journalctl@ceph.osd.3.smithi098.stdout:Mar 12 04:10:14 smithi098 podman[130860]: 2024-03-12 04:10:14.701372468 +0000 UTC m=+0.661781008 container died c9df55f4cd62248e7a6cadc7631f9a0890c25c6f610934192e8153b5a4b38232 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:15.219 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:14 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:14.963+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:15.336 DEBUG:teuthology.orchestra.run.smithi098:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.3.service' 2024-03-12T04:10:15.378 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:10:15.378 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-12T04:10:15.378 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-12T04:10:15.378 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.4 2024-03-12T04:10:15.646 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:15 smithi153 systemd[1]: Stopping Ceph osd.4 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:15.647 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:15 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:15.217+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:15.919 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:15 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:15.643+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:48.309270+0000 front 2024-03-12T04:09:48.309213+0000 (oldest deadline 2024-03-12T04:10:14.209114+0000) 2024-03-12T04:10:15.919 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:15 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:15.865+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:16.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:15 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:15.916+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:16.192 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:15 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:15.916+0000 7f78446f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:10:16.192 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:15 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:15.916+0000 7f78446f0700 -1 osd.4 160 *** Got signal Terminated *** 2024-03-12T04:10:16.193 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:15 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:15.916+0000 7f78446f0700 -1 osd.4 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:10:16.618 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:16 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:16.241+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:16.904 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:16 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:16.861+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:16.905 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:16 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:16.616+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:48.309270+0000 front 2024-03-12T04:09:48.309213+0000 (oldest deadline 2024-03-12T04:10:14.209114+0000) 2024-03-12T04:10:17.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:16 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:16.902+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:17.576 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:17.280+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:17.873 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:17.574+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:48.309270+0000 front 2024-03-12T04:09:48.309213+0000 (oldest deadline 2024-03-12T04:10:14.209114+0000) 2024-03-12T04:10:18.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:17.902+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:18.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:17 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:17.871+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:18.600 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:18.295+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:18.850 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:18.598+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:48.309270+0000 front 2024-03-12T04:09:48.309213+0000 (oldest deadline 2024-03-12T04:10:14.209114+0000) 2024-03-12T04:10:19.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:18.928+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:19.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:18 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:18.849+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:19.591 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:19.333+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:19.933 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:19.810+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:19.934 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:19.588+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:48.309270+0000 front 2024-03-12T04:09:48.309213+0000 (oldest deadline 2024-03-12T04:10:14.209114+0000) 2024-03-12T04:10:19.934 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:19.588+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.209544+0000 front 2024-03-12T04:09:54.209639+0000 (oldest deadline 2024-03-12T04:10:19.509565+0000) 2024-03-12T04:10:20.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:19 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:19.931+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:20.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:20.376+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:20.692 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:20.616+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:48.309270+0000 front 2024-03-12T04:09:48.309213+0000 (oldest deadline 2024-03-12T04:10:14.209114+0000) 2024-03-12T04:10:20.692 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4[78091]: 2024-03-12T04:10:20.616+0000 7f7840ce4700 -1 osd.4 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.209544+0000 front 2024-03-12T04:09:54.209639+0000 (oldest deadline 2024-03-12T04:10:19.509565+0000) 2024-03-12T04:10:21.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:20.887+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:21.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:20 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:20.835+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:21.692 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:21 smithi153 podman[101487]: 2024-03-12 04:10:21.450513978 +0000 UTC m=+5.877867122 container died e913f3621dd5181715ac8bbc996baf20cbf0561f8edfc970d901b112247e174c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-12T04:10:21.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:21 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:21.360+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:21.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:21 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:21.360+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:22.097 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:21 smithi153 podman[101487]: 2024-03-12 04:10:21.858442712 +0000 UTC m=+6.285795863 container cleanup e913f3621dd5181715ac8bbc996baf20cbf0561f8edfc970d901b112247e174c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-12T04:10:22.097 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:21 smithi153 bash[101487]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4 2024-03-12T04:10:22.097 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:21 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:21.863+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:22.097 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:21 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:21.834+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:22.097 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:21 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:21.834+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:22.372 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:22 smithi153 podman[101505]: 2024-03-12 04:10:22.093423314 +0000 UTC m=+0.637919764 container remove e913f3621dd5181715ac8bbc996baf20cbf0561f8edfc970d901b112247e174c (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-12T04:10:22.373 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:22.335+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:22.373 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:22.335+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:22.648 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:22 smithi153 podman[101599]: 2024-03-12 04:10:22.326544874 +0000 UTC m=+0.018697009 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:10:22.648 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:22 smithi153 podman[101599]: 2024-03-12 04:10:22.436018397 +0000 UTC m=+0.128170514 container create c30a40c811a862676cd4a7c483fcad93250e9e0b99546f2a6700d7d03df1094b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:10:22.649 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:22.371Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-03-12T04:10:22.649 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:22.371Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-03-12T04:10:22.649 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:22.374Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-03-12T04:10:22.649 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:22.374Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-03-12T04:10:22.649 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:22.374Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.98:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.98:8765: connect: connection refused" 2024-03-12T04:10:22.940 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:22.856+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:22.940 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:22.856+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:22.940 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:22.825+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:22.940 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:22 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:22.825+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:55.512218+0000 front 2024-03-12T04:09:55.512159+0000 (oldest deadline 2024-03-12T04:10:21.912324+0000) 2024-03-12T04:10:22.941 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:22 smithi153 podman[101599]: 2024-03-12 04:10:22.647092806 +0000 UTC m=+0.339244933 container init c30a40c811a862676cd4a7c483fcad93250e9e0b99546f2a6700d7d03df1094b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-12T04:10:22.941 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:22 smithi153 podman[101599]: 2024-03-12 04:10:22.650321221 +0000 UTC m=+0.342473337 container start c30a40c811a862676cd4a7c483fcad93250e9e0b99546f2a6700d7d03df1094b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-12T04:10:22.941 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:22 smithi153 podman[101599]: 2024-03-12 04:10:22.721910423 +0000 UTC m=+0.414062554 container attach c30a40c811a862676cd4a7c483fcad93250e9e0b99546f2a6700d7d03df1094b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:23.192 INFO:journalctl@ceph.osd.4.smithi153.stdout:Mar 12 04:10:22 smithi153 podman[101599]: 2024-03-12 04:10:22.939384774 +0000 UTC m=+0.631536893 container died c30a40c811a862676cd4a7c483fcad93250e9e0b99546f2a6700d7d03df1094b (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-4-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T04:10:23.558 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:23.298+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:23.560 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:23.298+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:23.571 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.4.service' 2024-03-12T04:10:23.616 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:10:23.616 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-12T04:10:23.616 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-12T04:10:23.616 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.5 2024-03-12T04:10:23.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:23.553Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T04:10:23.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:23.554Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T04:10:23.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:23 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:23.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T04:10:23.825 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:23 smithi153 systemd[1]: Stopping Ceph osd.5 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:24.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:23.836+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:24.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:23.836+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:24.193 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:23.823+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:24.193 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:23.823+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:55.512218+0000 front 2024-03-12T04:09:55.512159+0000 (oldest deadline 2024-03-12T04:10:21.912324+0000) 2024-03-12T04:10:24.193 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:23.938+0000 7f3e207bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:10:24.193 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:23.938+0000 7f3e207bf700 -1 osd.5 160 *** Got signal Terminated *** 2024-03-12T04:10:24.193 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:23 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:23.938+0000 7f3e207bf700 -1 osd.5 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:10:24.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:24.248+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:24.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:24.248+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:25.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:24.789+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:25.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:24.789+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:55.512218+0000 front 2024-03-12T04:09:55.512159+0000 (oldest deadline 2024-03-12T04:10:21.912324+0000) 2024-03-12T04:10:25.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:24.794+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:25.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:24 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:24.794+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:25.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:25.253+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:25.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:25.253+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:26.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:25.760+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:26.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:25.760+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:55.512218+0000 front 2024-03-12T04:09:55.512159+0000 (oldest deadline 2024-03-12T04:10:21.912324+0000) 2024-03-12T04:10:26.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:25.795+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:26.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:25 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:25.795+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:26.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:26.236+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:26.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:26.236+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:26.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:26.236+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:27.192 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:26.736+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:27.193 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:26.736+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:55.512218+0000 front 2024-03-12T04:09:55.512159+0000 (oldest deadline 2024-03-12T04:10:21.912324+0000) 2024-03-12T04:10:27.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:26.802+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:27.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:26 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:26.802+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:27.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:27.207+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:27.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:27.207+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:27.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:27.207+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:28.161 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:27.843+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:28.161 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:27.843+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:28.161 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:27.843+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:28.162 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:27.736+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:28.162 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:27.736+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:55.512218+0000 front 2024-03-12T04:09:55.512159+0000 (oldest deadline 2024-03-12T04:10:21.912324+0000) 2024-03-12T04:10:28.162 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:27 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:27.736+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:03.013221+0000 front 2024-03-12T04:10:03.013275+0000 (oldest deadline 2024-03-12T04:10:27.113063+0000) 2024-03-12T04:10:28.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:28.159+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:28.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:28.159+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:28.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:28.159+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:29.148 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:28.716+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.911236+0000 front 2024-03-12T04:09:47.911197+0000 (oldest deadline 2024-03-12T04:10:13.211148+0000) 2024-03-12T04:10:29.148 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:28.716+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:55.512218+0000 front 2024-03-12T04:09:55.512159+0000 (oldest deadline 2024-03-12T04:10:21.912324+0000) 2024-03-12T04:10:29.148 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5[82641]: 2024-03-12T04:10:28.716+0000 7f3e1cdb3700 -1 osd.5 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:03.013221+0000 front 2024-03-12T04:10:03.013275+0000 (oldest deadline 2024-03-12T04:10:27.113063+0000) 2024-03-12T04:10:29.149 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:28.839+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:29.149 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:28.839+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:29.149 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:28 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:28.839+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:29.433 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:29.146+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:29.434 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:29.146+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:29.434 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:29.146+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:29.692 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:29 smithi153 podman[101704]: 2024-03-12 04:10:29.432029992 +0000 UTC m=+5.618180290 container died 42f3b944dd81fc9f10d04d5d3a761074abbe01fe067deee60c9793858b6cac96 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-12T04:10:29.990 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:29 smithi153 podman[101704]: 2024-03-12 04:10:29.721032997 +0000 UTC m=+5.907183294 container cleanup 42f3b944dd81fc9f10d04d5d3a761074abbe01fe067deee60c9793858b6cac96 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:10:29.990 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:29 smithi153 bash[101704]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5 2024-03-12T04:10:29.991 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:29.875+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:29.991 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:29.875+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:29.991 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:29 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:29.875+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:30.352 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:29 smithi153 podman[101734]: 2024-03-12 04:10:29.989057385 +0000 UTC m=+0.558710275 container remove 42f3b944dd81fc9f10d04d5d3a761074abbe01fe067deee60c9793858b6cac96 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:10:30.353 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:30.106+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:30.353 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:30.106+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:30.353 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:30.106+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:30.636 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:30 smithi153 podman[101832]: 2024-03-12 04:10:30.254040496 +0000 UTC m=+0.018707238 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:10:30.636 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:30 smithi153 podman[101832]: 2024-03-12 04:10:30.361904287 +0000 UTC m=+0.126571084 container create 84a7c3ebd7c7594bf5ef8aac7e8f82ed7f0f1c4af02c88d37ae4766e485b6320 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-12T04:10:30.636 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:30 smithi153 podman[101832]: 2024-03-12 04:10:30.559229571 +0000 UTC m=+0.323896313 container init 84a7c3ebd7c7594bf5ef8aac7e8f82ed7f0f1c4af02c88d37ae4766e485b6320 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-12T04:10:30.636 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:30 smithi153 podman[101832]: 2024-03-12 04:10:30.562319659 +0000 UTC m=+0.326986406 container start 84a7c3ebd7c7594bf5ef8aac7e8f82ed7f0f1c4af02c88d37ae4766e485b6320 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-03-12T04:10:30.910 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:30 smithi153 podman[101832]: 2024-03-12 04:10:30.634084572 +0000 UTC m=+0.398751331 container attach 84a7c3ebd7c7594bf5ef8aac7e8f82ed7f0f1c4af02c88d37ae4766e485b6320 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:30.910 INFO:journalctl@ceph.osd.5.smithi153.stdout:Mar 12 04:10:30 smithi153 podman[101832]: 2024-03-12 04:10:30.857131398 +0000 UTC m=+0.621798141 container died 84a7c3ebd7c7594bf5ef8aac7e8f82ed7f0f1c4af02c88d37ae4766e485b6320 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:10:31.192 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:31.150+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:31.192 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:31.150+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:31.192 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:31.150+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:31.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:30.908+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:31.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:30.908+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:31.193 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:30 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:30.908+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:31.459 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.5.service' 2024-03-12T04:10:31.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:10:31.502 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-12T04:10:31.502 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-12T04:10:31.502 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6 2024-03-12T04:10:31.692 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:31 smithi153 systemd[1]: Stopping Ceph osd.6 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:32.147 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:31.816+0000 7f48ba59f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:31.816+0000 7f48ba59f700 -1 osd.6 160 *** Got signal Terminated *** 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:31.816+0000 7f48ba59f700 -1 osd.6 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:31.905+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:31.905+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:32.148 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:31 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:31.905+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:32.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:32.146+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:32.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:32.146+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:32.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:32.146+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:33.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:32.885+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:33.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:32.885+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:33.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:32 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:32.885+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:33.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:33.195+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:33.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:33.195+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:33.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:33.195+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:33.823 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:33.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T04:10:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:33.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T04:10:33.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:33 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:33.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T04:10:34.188 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:33.888+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:34.188 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:33.888+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:34.188 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:33 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:33.888+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:34.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:34.186+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:34.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:34.186+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:34.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:34.186+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:35.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:34.872+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:35.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:34.872+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:35.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:34 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:34.872+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:35.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:35.209+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:35.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:35.209+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:35.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:35.209+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:36.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:36.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:36.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:36.192 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:35 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:11.868353+0000 front 2024-03-12T04:10:11.868419+0000 (oldest deadline 2024-03-12T04:10:35.368215+0000) 2024-03-12T04:10:36.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:36.211+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:36.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:36.211+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:36.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:36.211+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:37.251 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:37 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:37.170+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:37.251 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:37 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:37.170+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:37.251 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:37 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:37.170+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: *** Caught signal (Segmentation fault) ** 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: in thread 7f48a1b69700 thread_name:safe_timer 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f48c19f5d20] 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55bb43ab3cca] 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55bb43b41b58] 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55bb43726da3] 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55bb439b2d01] 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55bb436a1974] 2024-03-12T04:10:37.252 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55bb436a39ca] 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55bb436a5933] 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55bb4360756a] 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x55bb43c61fef] 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55bb43c63311] 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48c19eb1ca] 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 13: clone() 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:36.953+0000 7f48a1b69700 -1 *** Caught signal (Segmentation fault) ** 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: in thread 7f48a1b69700 thread_name:safe_timer 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T04:10:37.253 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f48c19f5d20] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55bb43ab3cca] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55bb43b41b58] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55bb43726da3] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55bb439b2d01] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55bb436a1974] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55bb436a39ca] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55bb436a5933] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55bb4360756a] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x55bb43c61fef] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55bb43c63311] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48c19eb1ca] 2024-03-12T04:10:37.255 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 13: clone() 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -4227> 2024-03-12T04:10:12.026+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -4124> 2024-03-12T04:10:12.995+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -3996> 2024-03-12T04:10:13.949+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -3944> 2024-03-12T04:10:14.963+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -3854> 2024-03-12T04:10:15.916+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -3780> 2024-03-12T04:10:16.902+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -3734> 2024-03-12T04:10:17.871+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -3123> 2024-03-12T04:10:18.849+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -3071> 2024-03-12T04:10:19.810+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2551> 2024-03-12T04:10:20.835+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2395> 2024-03-12T04:10:21.834+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.256 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2394> 2024-03-12T04:10:21.834+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2322> 2024-03-12T04:10:22.856+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2321> 2024-03-12T04:10:22.856+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2283> 2024-03-12T04:10:23.836+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2282> 2024-03-12T04:10:23.836+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2236> 2024-03-12T04:10:24.794+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2235> 2024-03-12T04:10:24.794+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2176> 2024-03-12T04:10:25.795+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -2175> 2024-03-12T04:10:25.795+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1779> 2024-03-12T04:10:26.802+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1778> 2024-03-12T04:10:26.802+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1290> 2024-03-12T04:10:27.843+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1289> 2024-03-12T04:10:27.843+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.257 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1288> 2024-03-12T04:10:27.843+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1239> 2024-03-12T04:10:28.839+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1238> 2024-03-12T04:10:28.839+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1237> 2024-03-12T04:10:28.839+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1148> 2024-03-12T04:10:29.875+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1147> 2024-03-12T04:10:29.875+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1146> 2024-03-12T04:10:29.875+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1102> 2024-03-12T04:10:30.908+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1101> 2024-03-12T04:10:30.908+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1100> 2024-03-12T04:10:30.908+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1035> 2024-03-12T04:10:31.816+0000 7f48ba59f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1034> 2024-03-12T04:10:31.816+0000 7f48ba59f700 -1 osd.6 160 *** Got signal Terminated *** 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1032> 2024-03-12T04:10:31.816+0000 7f48ba59f700 -1 osd.6 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:10:37.258 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1029> 2024-03-12T04:10:31.905+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1028> 2024-03-12T04:10:31.905+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -1027> 2024-03-12T04:10:31.905+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -934> 2024-03-12T04:10:32.885+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -933> 2024-03-12T04:10:32.885+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -932> 2024-03-12T04:10:32.885+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -886> 2024-03-12T04:10:33.888+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -885> 2024-03-12T04:10:33.888+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.259 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -884> 2024-03-12T04:10:33.888+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -686> 2024-03-12T04:10:34.872+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -685> 2024-03-12T04:10:34.872+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -684> 2024-03-12T04:10:34.872+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -274> 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:47.865622+0000 front 2024-03-12T04:09:47.865607+0000 (oldest deadline 2024-03-12T04:10:11.965431+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -273> 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:51.965881+0000 front 2024-03-12T04:09:51.965898+0000 (oldest deadline 2024-03-12T04:10:21.366513+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -272> 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:01.366983+0000 front 2024-03-12T04:10:01.367099+0000 (oldest deadline 2024-03-12T04:10:27.266920+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: -271> 2024-03-12T04:10:35.823+0000 7f48b2b8b700 -1 osd.6 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:11.868353+0000 front 2024-03-12T04:10:11.868419+0000 (oldest deadline 2024-03-12T04:10:35.368215+0000) 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 0> 2024-03-12T04:10:36.953+0000 7f48a1b69700 -1 *** Caught signal (Segmentation fault) ** 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: in thread 7f48a1b69700 thread_name:safe_timer 2024-03-12T04:10:37.260 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f48c19f5d20] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x55bb43ab3cca] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x55bb43b41b58] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x55bb43726da3] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x55bb439b2d01] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x55bb436a1974] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x55bb436a39ca] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x55bb436a5933] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x55bb4360756a] 2024-03-12T04:10:37.261 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 10: (CommonSafeTimer::timer_thread()+0x13f) [0x55bb43c61fef] 2024-03-12T04:10:37.262 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 11: (CommonSafeTimerThread::entry()+0x11) [0x55bb43c63311] 2024-03-12T04:10:37.262 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48c19eb1ca] 2024-03-12T04:10:37.262 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 13: clone() 2024-03-12T04:10:37.262 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-12T04:10:37.262 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:36 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6[87599]: 2024-03-12T04:10:37.262 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:37 smithi153 podman[101938]: 2024-03-12 04:10:37.042611972 +0000 UTC m=+5.320199931 container died 23a1ba5d3665f5c70a40546fbfaa115d491682b5c78059c9ab4437c2de40fc54 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True) 2024-03-12T04:10:37.643 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:37 smithi153 podman[101938]: 2024-03-12 04:10:37.465993347 +0000 UTC m=+5.743581343 container cleanup 23a1ba5d3665f5c70a40546fbfaa115d491682b5c78059c9ab4437c2de40fc54 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-12T04:10:37.643 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:37 smithi153 bash[101938]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6 2024-03-12T04:10:37.942 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:37 smithi153 podman[101953]: 2024-03-12 04:10:37.642320067 +0000 UTC m=+0.601911480 container remove 23a1ba5d3665f5c70a40546fbfaa115d491682b5c78059c9ab4437c2de40fc54 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-12T04:10:37.942 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:37 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6.service: Main process exited, code=exited, status=139/n/a 2024-03-12T04:10:38.330 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:38.135+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:38.330 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:38.135+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:38.330 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:38 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:38.135+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:38.331 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:38 smithi153 podman[102047]: 2024-03-12 04:10:37.958106986 +0000 UTC m=+0.018431368 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:10:38.331 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:38 smithi153 podman[102047]: 2024-03-12 04:10:38.059906746 +0000 UTC m=+0.120231114 container create 5bb24a0cc119a747b45a95ff56ec98d8dab9c11ab657e3efee8943e5b73d2ce0 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-12T04:10:38.331 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:38 smithi153 podman[102047]: 2024-03-12 04:10:38.262676666 +0000 UTC m=+0.323001044 container init 5bb24a0cc119a747b45a95ff56ec98d8dab9c11ab657e3efee8943e5b73d2ce0 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-12T04:10:38.331 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:38 smithi153 podman[102047]: 2024-03-12 04:10:38.266173957 +0000 UTC m=+0.326498327 container start 5bb24a0cc119a747b45a95ff56ec98d8dab9c11ab657e3efee8943e5b73d2ce0 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-12T04:10:38.331 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:38 smithi153 podman[102047]: 2024-03-12 04:10:38.328953446 +0000 UTC m=+0.389277823 container attach 5bb24a0cc119a747b45a95ff56ec98d8dab9c11ab657e3efee8943e5b73d2ce0 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-03-12T04:10:38.692 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:38 smithi153 podman[102047]: 2024-03-12 04:10:38.568382822 +0000 UTC m=+0.628707208 container died 5bb24a0cc119a747b45a95ff56ec98d8dab9c11ab657e3efee8943e5b73d2ce0 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-12T04:10:39.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6.service' 2024-03-12T04:10:39.102 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:39 smithi153 podman[102073]: 2024-03-12 04:10:39.057146057 +0000 UTC m=+0.483196273 container remove 5bb24a0cc119a747b45a95ff56ec98d8dab9c11ab657e3efee8943e5b73d2ce0 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:39.102 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:39 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6.service: Failed with result 'exit-code'. 2024-03-12T04:10:39.102 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:39 smithi153 systemd[1]: Stopped Ceph osd.6 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:10:39.102 INFO:journalctl@ceph.osd.6.smithi153.stdout:Mar 12 04:10:39 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.6.service: Consumed 4.352s CPU time. 2024-03-12T04:10:39.118 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:10:39.118 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-12T04:10:39.118 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-12T04:10:39.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.7 2024-03-12T04:10:39.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:39.123+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:39.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:39.123+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:39.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:39.123+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:39.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:39.123+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:12.811910+0000 front 2024-03-12T04:10:12.811964+0000 (oldest deadline 2024-03-12T04:10:38.711817+0000) 2024-03-12T04:10:39.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 systemd[1]: Stopping Ceph osd.7 for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:39.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:39.443+0000 7fc4aca7f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-12T04:10:39.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:39.443+0000 7fc4aca7f700 -1 osd.7 160 *** Got signal Terminated *** 2024-03-12T04:10:39.942 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:39 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:39.443+0000 7fc4aca7f700 -1 osd.7 160 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-12T04:10:40.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:40.103+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:40.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:40.103+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:40.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:40.103+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:40.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:40 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:40.103+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:12.811910+0000 front 2024-03-12T04:10:12.811964+0000 (oldest deadline 2024-03-12T04:10:38.711817+0000) 2024-03-12T04:10:41.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:41.065+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:41.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:41.065+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:41.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:41.065+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:41.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:41 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:41.065+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:12.811910+0000 front 2024-03-12T04:10:12.811964+0000 (oldest deadline 2024-03-12T04:10:38.711817+0000) 2024-03-12T04:10:42.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:42.019+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:42.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:42.019+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:42.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:42.019+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:42.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:42 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:42.019+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:12.811910+0000 front 2024-03-12T04:10:12.811964+0000 (oldest deadline 2024-03-12T04:10:38.711817+0000) 2024-03-12T04:10:43.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:43.018+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:43.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:43.018+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:43.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:43.018+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:43.443 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:43 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:43.018+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:12.811910+0000 front 2024-03-12T04:10:12.811964+0000 (oldest deadline 2024-03-12T04:10:38.711817+0000) 2024-03-12T04:10:43.823 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:43.554Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=5 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T04:10:43.823 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:43.555Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi098.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.98:8443: connect: connection refused" 2024-03-12T04:10:43.824 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:43 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:43.556Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi153.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.153:8443: connect: connection refused" 2024-03-12T04:10:44.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:44.048+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6806 osd.0 since back 2024-03-12T04:09:49.008353+0000 front 2024-03-12T04:09:49.008278+0000 (oldest deadline 2024-03-12T04:10:14.808470+0000) 2024-03-12T04:10:44.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:44.048+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6814 osd.1 since back 2024-03-12T04:09:54.808983+0000 front 2024-03-12T04:09:54.809039+0000 (oldest deadline 2024-03-12T04:10:20.609433+0000) 2024-03-12T04:10:44.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:44.048+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6822 osd.2 since back 2024-03-12T04:10:00.610064+0000 front 2024-03-12T04:10:00.609977+0000 (oldest deadline 2024-03-12T04:10:25.910093+0000) 2024-03-12T04:10:44.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:44 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7[92150]: 2024-03-12T04:10:44.048+0000 7fc4a506b700 -1 osd.7 160 heartbeat_check: no reply from 172.21.15.98:6830 osd.3 since back 2024-03-12T04:10:12.811910+0000 front 2024-03-12T04:10:12.811964+0000 (oldest deadline 2024-03-12T04:10:38.711817+0000) 2024-03-12T04:10:45.037 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:44 smithi153 podman[102152]: 2024-03-12 04:10:44.768937955 +0000 UTC m=+5.433018336 container died 782d8be774e8d57fdb4e6315efd54de04389065c5c1f56a5fb2eee5784bb83a3 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-12T04:10:45.371 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:45 smithi153 podman[102152]: 2024-03-12 04:10:45.135121065 +0000 UTC m=+5.799201446 container cleanup 782d8be774e8d57fdb4e6315efd54de04389065c5c1f56a5fb2eee5784bb83a3 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-12T04:10:45.371 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:45 smithi153 bash[102152]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7 2024-03-12T04:10:45.692 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:45 smithi153 podman[102170]: 2024-03-12 04:10:45.369561535 +0000 UTC m=+0.592050165 container remove 782d8be774e8d57fdb4e6315efd54de04389065c5c1f56a5fb2eee5784bb83a3 (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:46.024 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:45 smithi153 podman[102265]: 2024-03-12 04:10:45.606077994 +0000 UTC m=+0.018592465 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-12T04:10:46.024 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:45 smithi153 podman[102265]: 2024-03-12 04:10:45.703615764 +0000 UTC m=+0.116130222 container create 717b2de82558bb59561bfefc167295785169472903480517fa497433d43563ee (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-12T04:10:46.024 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:45 smithi153 podman[102265]: 2024-03-12 04:10:45.939413438 +0000 UTC m=+0.351927897 container init 717b2de82558bb59561bfefc167295785169472903480517fa497433d43563ee (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-12T04:10:46.024 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:45 smithi153 podman[102265]: 2024-03-12 04:10:45.943095325 +0000 UTC m=+0.355609783 container start 717b2de82558bb59561bfefc167295785169472903480517fa497433d43563ee (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:46.025 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:46 smithi153 podman[102265]: 2024-03-12 04:10:46.02269784 +0000 UTC m=+0.435212309 container attach 717b2de82558bb59561bfefc167295785169472903480517fa497433d43563ee (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-12T04:10:46.442 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:46 smithi153 podman[102265]: 2024-03-12 04:10:46.228852411 +0000 UTC m=+0.641366871 container died 717b2de82558bb59561bfefc167295785169472903480517fa497433d43563ee (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-12T04:10:46.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.7.service' 2024-03-12T04:10:46.786 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:46 smithi153 podman[102291]: 2024-03-12 04:10:46.734560853 +0000 UTC m=+0.496797118 container remove 717b2de82558bb59561bfefc167295785169472903480517fa497433d43563ee (image=quay.ceph.io/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-osd-7-deactivate, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD) 2024-03-12T04:10:46.786 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:46 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-12T04:10:46.786 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:46 smithi153 systemd[1]: Stopped Ceph osd.7 for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:10:46.786 INFO:journalctl@ceph.osd.7.smithi153.stdout:Mar 12 04:10:46 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@osd.7.service: Consumed 4.361s CPU time. 2024-03-12T04:10:46.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:10:46.799 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-12T04:10:46.799 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-12T04:10:46.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a 2024-03-12T04:10:47.122 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:46 smithi153 systemd[1]: Stopping Ceph prometheus.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.120Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-12T04:10:47.430 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.121Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-12T04:10:47.431 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.122Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-12T04:10:47.431 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.122Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-12T04:10:47.431 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a[64232]: ts=2024-03-12T04:10:47.122Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-12T04:10:47.431 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 podman[102371]: 2024-03-12 04:10:47.13087229 +0000 UTC m=+0.108814155 container died 7597e44b0bb5de73778cb090d5517f781735e2e0f132087f9f9bffc8a47eb8e8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:47.431 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 podman[102371]: 2024-03-12 04:10:47.428645077 +0000 UTC m=+0.406586946 container cleanup 7597e44b0bb5de73778cb090d5517f781735e2e0f132087f9f9bffc8a47eb8e8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:47.431 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 bash[102371]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a 2024-03-12T04:10:47.692 INFO:journalctl@ceph.prometheus.a.smithi153.stdout:Mar 12 04:10:47 smithi153 podman[102385]: 2024-03-12 04:10:47.604655267 +0000 UTC m=+0.476008630 container remove 7597e44b0bb5de73778cb090d5517f781735e2e0f132087f9f9bffc8a47eb8e8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:47.851 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-12T04:10:47.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-12T04:10:47.890 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-12T04:10:47.890 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T04:10:49.627 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:49 smithi098 systemd[1]: Stopping Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:49.883 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:49 smithi098 podman[131266]: 2024-03-12 04:10:49.62505623 +0000 UTC m=+0.110580722 container died ba4b1fd77bbb231b13261a57efb1ab9cddcc6ded013bc47c9ed7818b95d840de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:50.258 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:49 smithi098 podman[131266]: 2024-03-12 04:10:49.989738247 +0000 UTC m=+0.475262741 container cleanup ba4b1fd77bbb231b13261a57efb1ab9cddcc6ded013bc47c9ed7818b95d840de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:50.259 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:49 smithi098 bash[131266]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a 2024-03-12T04:10:50.573 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:50 smithi098 podman[131280]: 2024-03-12 04:10:50.257423137 +0000 UTC m=+0.636638741 container remove ba4b1fd77bbb231b13261a57efb1ab9cddcc6ded013bc47c9ed7818b95d840de (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:50.573 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:50 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T04:10:50.573 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:50 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-12T04:10:50.573 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:50 smithi098 systemd[1]: Stopped Ceph node-exporter.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:10:50.574 INFO:journalctl@ceph.node-exporter.a.smithi098.stdout:Mar 12 04:10:50 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 3.016s CPU time. 2024-03-12T04:10:50.869 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:50 smithi098 systemd[1]: Stopping Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:10:50.869 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:50 smithi098 ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a[79517]: ts=2024-03-12T04:10:50.826Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-12T04:10:51.155 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:50 smithi098 podman[131385]: 2024-03-12 04:10:50.867779553 +0000 UTC m=+0.147308562 container stop 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:51.155 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:50 smithi098 podman[131385]: 2024-03-12 04:10:50.893562356 +0000 UTC m=+0.173091364 container died 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:51.413 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:51 smithi098 podman[131385]: 2024-03-12 04:10:51.252248263 +0000 UTC m=+0.531777272 container cleanup 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:51.413 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:51 smithi098 bash[131385]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a 2024-03-12T04:10:51.707 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:51 smithi098 podman[131399]: 2024-03-12 04:10:51.412139385 +0000 UTC m=+0.535110634 container remove 5fd1b9f8e512dfa5adfe7b8b91ad8d530dbc3e2ce48f9ad6e74052639ca6e5c7 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-12T04:10:51.707 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:51 smithi098 podman[131399]: 2024-03-12 04:10:51.479349997 +0000 UTC m=+0.602321243 volume remove 2758b10a68e464d14a66a36f64df6422801ad2f52ee17db5faae5813dc450031 2024-03-12T04:10:52.073 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:51 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-12T04:10:52.073 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:51 smithi098 systemd[1]: Stopped Ceph alertmanager.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:10:52.073 INFO:journalctl@ceph.alertmanager.a.smithi098.stdout:Mar 12 04:10:51 smithi098 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.514s CPU time. 2024-03-12T04:11:14.921 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-12T04:11:16.659 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:16 smithi153 systemd[1]: Stopping Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:11:16.942 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:16 smithi153 podman[102781]: 2024-03-12 04:11:16.656482864 +0000 UTC m=+0.107492627 container died a0fe78879ec55c03c79c8b0c78a0f7a9f10e99eec464d01e2e83d876c2cbe19c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T04:11:17.340 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:17 smithi153 podman[102781]: 2024-03-12 04:11:17.104889618 +0000 UTC m=+0.555899374 container cleanup a0fe78879ec55c03c79c8b0c78a0f7a9f10e99eec464d01e2e83d876c2cbe19c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T04:11:17.340 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:17 smithi153 bash[102781]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b 2024-03-12T04:11:17.648 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:17 smithi153 podman[102795]: 2024-03-12 04:11:17.339298189 +0000 UTC m=+0.686897352 container remove a0fe78879ec55c03c79c8b0c78a0f7a9f10e99eec464d01e2e83d876c2cbe19c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-12T04:11:17.648 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:17 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-12T04:11:17.648 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:17 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-12T04:11:17.648 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:17 smithi153 systemd[1]: Stopped Ceph node-exporter.b for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:11:17.648 INFO:journalctl@ceph.node-exporter.b.smithi153.stdout:Mar 12 04:11:17 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 3.059s CPU time. 2024-03-12T04:11:17.904 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:17 smithi153 systemd[1]: Stopping Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715... 2024-03-12T04:11:18.161 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:17 smithi153 podman[102903]: 2024-03-12 04:11:17.902947153 +0000 UTC m=+0.118451201 container died 499a74c577ce46021f3273f2a6d6114c5583b4e005d2e635bf053f253cd72f90 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, version=8.5, architecture=x86_64, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, release=239.1651231664) 2024-03-12T04:11:18.442 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:18 smithi153 podman[102903]: 2024-03-12 04:11:18.233472535 +0000 UTC m=+0.448976583 container cleanup 499a74c577ce46021f3273f2a6d6114c5583b4e005d2e635bf053f253cd72f90 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, io.openshift.expose-services=, version=8.5, maintainer=Paul Cuzner , name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base rhel8, architecture=x86_64, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-03-12T04:11:18.442 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:18 smithi153 bash[102903]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a 2024-03-12T04:11:18.442 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:18 smithi153 podman[102917]: 2024-03-12 04:11:18.401321602 +0000 UTC m=+0.497766426 container remove 499a74c577ce46021f3273f2a6d6114c5583b4e005d2e635bf053f253cd72f90 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-ac0aaff8-e020-11ee-95c8-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, version=8.5, name=ubi8, release=239.1651231664, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-12T04:11:18.442 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:18 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-12T04:11:18.942 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:18 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-12T04:11:18.942 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:18 smithi153 systemd[1]: Stopped Ceph grafana.a for ac0aaff8-e020-11ee-95c8-87774f69a715. 2024-03-12T04:11:18.942 INFO:journalctl@ceph.grafana.a.smithi153.stdout:Mar 12 04:11:18 smithi153 systemd[1]: ceph-ac0aaff8-e020-11ee-95c8-87774f69a715@grafana.a.service: Consumed 9.012s CPU time. 2024-03-12T04:11:30.281 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T04:11:30.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-12T04:11:30.345 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-12T04:11:30.347 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/remote/smithi098/crash 2024-03-12T04:11:30.348 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/crash -- . 2024-03-12T04:11:30.393 INFO:teuthology.orchestra.run.smithi098.stderr:tar: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T04:11:30.394 INFO:teuthology.orchestra.run.smithi098.stderr:tar: Error is not recoverable: exiting now 2024-03-12T04:11:30.396 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/remote/smithi153/crash 2024-03-12T04:11:30.397 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/crash -- . 2024-03-12T04:11:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-12T04:11:30.436 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-03-12T04:11:30.437 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-12T04:11:30.437 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T04:11:30.469 INFO:teuthology.orchestra.run.smithi098.stdout:2024-03-12T03:57:51.392089+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-12T04:11:30.469 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-12T04:11:30.470 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[SEC\]' /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T04:11:30.539 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[ERR\]' /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T04:11:30.609 DEBUG:teuthology.orchestra.run.smithi098:> sudo egrep '\[WRN\]' /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-03-12T04:11:30.678 INFO:teuthology.orchestra.run.smithi098.stdout:2024-03-12T03:57:51.392089+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-12T04:11:30.678 INFO:tasks.cephadm:Compressing logs... 2024-03-12T04:11:30.678 DEBUG:teuthology.orchestra.run.smithi098:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T04:11:30.721 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-12T04:11:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T04:11:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:: No such file or directory 2024-03-12T04:11:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-12T04:11:30.747 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T04:11:30.748 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi098.tgzbnw.log 2024-03-12T04:11:30.752 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.a.log: /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-12T04:11:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-12T04:11:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi098.tgzbnw.log: /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose 76.8% -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mgr.y.log -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi098.tgzbnw.log.gz 2024-03-12T04:11:30.753 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T04:11:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-12T04:11:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-12T04:11:30.753 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T04:11:30.753 INFO:teuthology.orchestra.run.smithi098.stderr: 90.3% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.6.log: /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.4.log: /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.cephadm.log: 91.6% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T04:11:30.755 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T04:11:30.756 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.0.log: 94.3% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T04:11:30.757 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log: 91.0% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-12T04:11:30.757 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-12T04:11:30.757 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-volume.log: 89.1% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T04:11:30.758 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi153.tuboyg.log 2024-03-12T04:11:30.763 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log 2024-03-12T04:11:30.764 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-12T04:11:30.764 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T04:11:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.b.log: 86.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-12T04:11:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log: 94.8% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-12T04:11:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-12T04:11:30.766 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-12T04:11:30.770 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-volume.log: 93.9% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-12T04:11:30.771 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi153.tuboyg.log: 75.9% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-client.rgw.foo.smithi153.tuboyg.log.gz 2024-03-12T04:11:30.772 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T04:11:30.782 INFO:teuthology.orchestra.run.smithi098.stderr:/var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-12T04:11:30.896 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-12T04:11:31.239 INFO:teuthology.orchestra.run.smithi098.stderr: 88.9% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-12T04:11:31.277 INFO:teuthology.orchestra.run.smithi153.stderr: 92.5% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-12T04:11:31.598 INFO:teuthology.orchestra.run.smithi098.stderr: 92.3% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-12T04:11:32.209 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-12T04:11:32.381 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-12T04:11:32.579 INFO:teuthology.orchestra.run.smithi153.stderr: 93.9% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-12T04:11:32.605 INFO:teuthology.orchestra.run.smithi098.stderr: 91.0% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-12T04:11:32.710 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-12T04:11:32.742 INFO:teuthology.orchestra.run.smithi153.stderr: 93.4% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-12T04:11:32.814 INFO:teuthology.orchestra.run.smithi153.stderr: 93.6% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-12T04:11:32.816 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-03-12T04:11:32.816 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m2.080s 2024-03-12T04:11:32.816 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m7.932s 2024-03-12T04:11:32.816 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.314s 2024-03-12T04:11:32.943 INFO:teuthology.orchestra.run.smithi098.stderr: 93.5% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-12T04:11:33.056 INFO:teuthology.orchestra.run.smithi098.stderr: 93.4% -- replaced with /var/log/ceph/ac0aaff8-e020-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-12T04:11:33.058 INFO:teuthology.orchestra.run.smithi098.stderr: 2024-03-12T04:11:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:real 0m2.324s 2024-03-12T04:11:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:user 0m10.725s 2024-03-12T04:11:33.058 INFO:teuthology.orchestra.run.smithi098.stderr:sys 0m0.416s 2024-03-12T04:11:33.059 INFO:tasks.cephadm:Archiving logs... 2024-03-12T04:11:33.059 DEBUG:teuthology.misc:Transferring archived files from smithi098:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/remote/smithi098/log 2024-03-12T04:11:33.065 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T04:11:34.535 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/remote/smithi153/log 2024-03-12T04:11:34.536 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-12T04:11:35.721 INFO:tasks.cephadm:Removing cluster... 2024-03-12T04:11:35.722 DEBUG:teuthology.orchestra.run.smithi098:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 --force 2024-03-12T04:11:36.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid ac0aaff8-e020-11ee-95c8-87774f69a715 --force 2024-03-12T04:11:36.310 INFO:tasks.cephadm:Removing cephadm ... 2024-03-12T04:11:36.310 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T04:11:36.325 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-12T04:11:36.339 INFO:tasks.cephadm:Teardown complete 2024-03-12T04:11:36.340 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-12T04:11:36.350 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-12T04:11:36.350 DEBUG:teuthology.orchestra.run.smithi098:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:11:36.367 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-12T04:11:36.378 INFO:teuthology.orchestra.run.smithi098.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:11:36.383 INFO:teuthology.orchestra.run.smithi098.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:11:36.383 INFO:teuthology.orchestra.run.smithi098.stdout:=============================================================================== 2024-03-12T04:11:36.384 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 -641us[ -641us] +/- 95ms 2024-03-12T04:11:36.384 INFO:teuthology.orchestra.run.smithi098.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -1519us[-1501us] +/- 52ms 2024-03-12T04:11:36.384 INFO:teuthology.orchestra.run.smithi098.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +2291us[+2309us] +/- 67ms 2024-03-12T04:11:36.384 INFO:teuthology.orchestra.run.smithi098.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:11:36.392 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-03-12T04:11:36.397 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-12T04:11:36.397 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-03-12T04:11:36.397 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -648us[ -648us] +/- 95ms 2024-03-12T04:11:36.397 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv02.front.sepia.ceph.com 3 6 377 27 -1514us[-1495us] +/- 52ms 2024-03-12T04:11:36.397 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +2299us[+2299us] +/- 67ms 2024-03-12T04:11:36.398 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-12T04:11:36.398 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-12T04:11:36.408 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-12T04:11:36.409 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-12T04:11:36.437 DEBUG:teuthology.orchestra.run.smithi098:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T04:11:36.528 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-12T04:11:36.610 DEBUG:teuthology.orchestra.run.smithi098:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T04:11:36.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:11:36.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-12T04:11:36.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:11:36.673 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-12T04:11:36.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-12T04:11:36.706 INFO:teuthology.task.internal:Duration was 2936.452643 seconds 2024-03-12T04:11:36.707 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-12T04:11:36.740 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-12T04:11:36.740 DEBUG:teuthology.orchestra.run.smithi098:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T04:11:36.743 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-12T04:11:36.783 INFO:teuthology.orchestra.run.smithi098.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:11:36.786 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-12T04:11:37.110 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-12T04:11:37.110 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi098.front.sepia.ceph.com 2024-03-12T04:11:37.110 DEBUG:teuthology.orchestra.run.smithi098:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T04:11:37.169 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-03-12T04:11:37.169 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-12T04:11:37.200 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-12T04:11:37.200 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T04:11:37.212 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-12T04:11:37.269 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-12T04:11:37.269 DEBUG:teuthology.orchestra.run.smithi098:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T04:11:37.280 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-12T04:11:37.843 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-12T04:11:37.853 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-12T04:11:37.854 DEBUG:teuthology.orchestra.run.smithi098:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T04:11:37.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-12T04:11:37.908 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-12T04:11:37.919 DEBUG:teuthology.orchestra.run.smithi098:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:11:37.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:11:37.946 INFO:teuthology.orchestra.run.smithi098.stdout:kernel.core_pattern = core 2024-03-12T04:11:37.974 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-03-12T04:11:38.003 DEBUG:teuthology.orchestra.run.smithi098:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:11:38.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:11:38.050 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-12T04:11:38.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-12T04:11:38.064 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-12T04:11:38.075 INFO:teuthology.task.internal:Transferring archived files... 2024-03-12T04:11:38.075 DEBUG:teuthology.misc:Transferring archived files from smithi098:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/remote/smithi098 2024-03-12T04:11:38.076 DEBUG:teuthology.orchestra.run.smithi098:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T04:11:38.144 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883/remote/smithi153 2024-03-12T04:11:38.145 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-12T04:11:38.227 INFO:teuthology.task.internal:Removing archive directory... 2024-03-12T04:11:38.228 DEBUG:teuthology.orchestra.run.smithi098:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T04:11:38.231 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-12T04:11:38.246 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-12T04:11:38.282 INFO:teuthology.task.internal:Not uploading archives. 2024-03-12T04:11:38.283 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-12T04:11:38.359 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-12T04:11:38.360 DEBUG:teuthology.orchestra.run.smithi098:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T04:11:38.362 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-12T04:11:38.374 INFO:teuthology.orchestra.run.smithi098.stdout: 265839 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 04:11 /home/ubuntu/cephtest 2024-03-12T04:11:38.376 INFO:teuthology.orchestra.run.smithi153.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 12 04:11 /home/ubuntu/cephtest 2024-03-12T04:11:38.377 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-12T04:11:38.398 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-12T04:11:38.553 INFO:teuthology.nuke:Checking targets against current locks 2024-03-12T04:11:38.583 DEBUG:teuthology.nuke:shortname: smithi098 2024-03-12T04:11:38.583 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T04:11:38.607 DEBUG:teuthology.nuke:shortname: smithi153 2024-03-12T04:11:38.607 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-12T04:11:38.610 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi098.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:11:37.565672', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBORZAmOWZtokeQB+CMilfY5yhRq4Gplc3CKGrMPgUPGx9q7vU9VWplWnjhKre61wRsoBsHOmSZEIJNdERvKSIPU='} 2024-03-12T04:11:38.634 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591883', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-12 03:11:37.564412', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCBS8D6UZq9/Mh1qPo4j827Ceqdn5bL1PWvfMLfF1ULEefVUd9uWOx65j2krI3URLO/cZi1Ijd7i7L0lIaNMOx8='} 2024-03-12T04:11:38.664 INFO:teuthology.orchestra.console.smithi098:Power off 2024-03-12T04:11:38.664 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T04:11:38.688 DEBUG:teuthology.orchestra.console.smithi098:power off output: Chassis Power Control: Down/Off 2024-03-12T04:11:38.688 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:11:38.706 INFO:teuthology.orchestra.console.smithi153:Power off 2024-03-12T04:11:38.707 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-12T04:11:38.715 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-12T04:11:38.729 DEBUG:teuthology.orchestra.console.smithi153:power off output: Chassis Power Control: Down/Off 2024-03-12T04:11:38.729 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:11:38.753 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-12T04:11:42.716 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:11:42.754 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:11:42.826 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is on 2024-03-12T04:11:42.868 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is on 2024-03-12T04:11:46.830 DEBUG:teuthology.orchestra.console.smithi098:pexpect command: ipmitool -H smithi098.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:11:46.870 DEBUG:teuthology.orchestra.console.smithi153:pexpect command: ipmitool -H smithi153.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-12T04:11:46.941 DEBUG:teuthology.orchestra.console.smithi098:check power output: Chassis Power is off 2024-03-12T04:11:46.980 DEBUG:teuthology.orchestra.console.smithi153:check power output: Chassis Power is off 2024-03-12T04:11:47.042 INFO:teuthology.orchestra.console.smithi098:Power off completed 2024-03-12T04:11:47.081 INFO:teuthology.orchestra.console.smithi153:Power off completed 2024-03-12T04:11:47.185 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2936.4526426792145 failure_reason: '"2024-03-12T03:57:51.392089+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-03-12T04:11:47.185 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-12T04:11:47.261 INFO:teuthology.run:FAIL